Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-15 Thread Jitendra Yadav
Hi, Looks like you got some pace, did you also tried with compression parameter? I think you will get more optimization with it. Also file transfer speed depends on our network bandwidth between PNN/SNN and network traffic b/w nodes.What's your network conf? Thanks On Wed, Aug 14, 2013 at 11:39

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-15 Thread lei liu
Hi Jitendra, I don't use the compression parameter. My network card is 100M/s, and I set the dfs.image.transfer.bandwidthPerSec. to 50M, so I think Active NameNode still has 50M bandwidth to be used to handle RPC request, why the OPS dropped by 50%？ 2013/8/15 Jitendra Yadav

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-13 Thread lei liu
The fsimage file size is 1658934155 2013/8/13 Harsh J ha...@cloudera.com How large are your checkpointed fsimage files? On Mon, Aug 12, 2013 at 3:42 PM, lei liu liulei...@gmail.com wrote: When Standby Namenode is doing checkpoint, upload the image file to Active NameNode, the Active

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-13 Thread lei liu
I write one programm to test NameNode performance. Please see the EditLogPerformance.java I use 60 threads to execute the EditLogPerformance.javacode, the testing result is below content: 2013-08-13 17:43:01,479 INFO my.EditLogPerformance (EditLogPerformance.java:run(37)) - totalCount:10392810

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-13 Thread Harsh J
Perhaps turning on fsimage compression may help. See documentation of dfs.image.compress at http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml. You can also try to throttle the bandwidth it uses via dfs.image.transfer.bandwidthPerSec. On Tue, Aug 13, 2013 at

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-13 Thread Jitendra Yadav
Hi, I'm agreed with Harsh comment on image file compression and transfer bandwidth parameter for optimizing checkpoint process. In addition I'm not able to correlate your performance program log timings(less then 10) and file transfer logs timing on active/stand by nodes. Thanks On Tue, Aug 13,

when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-12 Thread lei liu
When Standby Namenode is doing checkpoint, upload the image file to Active NameNode, the Active NameNode is very slow. What is reason result to the Active NameNode is slow? Thanks, LiuLei

Re: when Standby Namenode is doing checkpoint, the Active NameNode is slow.

2013-08-12 Thread Harsh J
How large are your checkpointed fsimage files? On Mon, Aug 12, 2013 at 3:42 PM, lei liu liulei...@gmail.com wrote: When Standby Namenode is doing checkpoint, upload the image file to Active NameNode, the Active NameNode is very slow. What is reason result to the Active NameNode is slow?