[
https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218813#comment-13218813
]
Todd Lipcon commented on HDFS-1623:
-----------------------------------
With HDFS-3020, HDFS-3023, HDFS-3024, HDFS-3025, I got the following timings:
{code}
teragen-4MB-block-ha-off-1.txt: Total time spent by all maps in
occupied slots (ms)=255005244
teragen-4MB-block-ha-off-2.txt: Total time spent by all maps in
occupied slots (ms)=248092620
teragen-4MB-block-ha-off-3.txt: Total time spent by all maps in
occupied slots (ms)=256926353
teragen-4MB-block-ha-off-4.txt: Total time spent by all maps in
occupied slots (ms)=244320729
teragen-4MB-block-ha-off-5.txt: Total time spent by all maps in
occupied slots (ms)=248901067
teragen-4MB-block-ha-off-6.txt: Total time spent by all maps in
occupied slots (ms)=234409970
teragen-4MB-block-ha-off-7.txt: Total time spent by all maps in
occupied slots (ms)=224624077
teragen-4MB-block-ha-off-8.txt: Total time spent by all maps in
occupied slots (ms)=235166437
----
teragen-4MB-block-trunk-1.txt: Total time spent by all maps in
occupied slots (ms)=247575318
teragen-4MB-block-trunk-2.txt: Total time spent by all maps in
occupied slots (ms)=234090512
teragen-4MB-block-trunk-3.txt: Total time spent by all maps in
occupied slots (ms)=241264032
teragen-4MB-block-trunk-4.txt: Total time spent by all maps in
occupied slots (ms)=242941073
teragen-4MB-block-trunk-5.txt: Total time spent by all maps in
occupied slots (ms)=236123386
teragen-4MB-block-trunk-6.txt: Total time spent by all maps in
occupied slots (ms)=243662148
teragen-4MB-block-trunk-7.txt: Total time spent by all maps in
occupied slots (ms)=240128084
teragen-4MB-block-trunk-8.txt: Total time spent by all maps in
occupied slots (ms)=220212020
{code}
I ran a t-test which says that the difference in means isn't statistically
significant. I'm also running the 256M-block teragen just to be safe. It's not
complete yet but so far the results look good.
The optimizations also reduced the edit log size for the 4MB-block by a factor
of two.
So I think once these above JIRAs are committed, we should be fine to merge to
trunk. I'll also continue to work on the performance with HA on, but the
important issue for merge is to make sure we don't regress the non-HA case.
> High Availability Framework for HDFS NN
> ---------------------------------------
>
> Key: HDFS-1623
> URL: https://issues.apache.org/jira/browse/HDFS-1623
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Sanjay Radia
> Assignee: Sanjay Radia
> Attachments: HA-tests.pdf, HDFS-1623.trunk.patch,
> HDFS-High-Availability.pdf, NameNode HA_v2.pdf, NameNode HA_v2_1.pdf,
> Namenode HA Framework.pdf, dfsio-results.tsv, ha-testplan.pdf, ha-testplan.tex
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
