Re: why my test result on dfs short circuit read is slower?

2013-02-18 Thread Michael Segel
On Feb 17, 2013, at 7:09 PM, Liu, Raymond raymond@intel.com wrote: io.file.buffer.size Drop this down to 64KB not 128KB. You have 16 cpu which really means 8 cores and 4 disks. Do you have Ganglia up and running? I'll wager that you'll see a lot of wait cpu cycles in both cases.

RE: why my test result on dfs short circuit read is slower?

2013-02-17 Thread Liu, Raymond
] Sent: Saturday, February 16, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs short circuit read is slower? Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short

RE: why my test result on dfs short circuit read is slower?

2013-02-17 Thread Liu, Raymond
, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs short circuit read is slower? Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true

答复: why my test result on dfs short circuit read is slower?

2013-02-17 Thread 谢良
[raymond@intel.com] 发送时间: 2013年2月18日 14:04 收件人: user@hadoop.apache.org 主题: RE: why my test result on dfs short circuit read is slower? Alright, I think in my sequence read scenario, it is possible that shortcircuit read is actually slower than read through datanode. For, when read through datanode

why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi I tried to use short circuit read to improve my hbase cluster MR scan performance. I have the following setting in hdfs-site.xml dfs.client.read.shortcircuit set to true dfs.block.local-path-access.user set to MR job runner. The cluster is 1+4 node

答复: why my test result on dfs short circuit read is slower?

2013-02-15 Thread 谢良
Hi Raymond, did you enable security feature in your cluster? there'll be no obvious benefit be found if so. Regards, Liang ___ 发件人: Liu, Raymond [raymond@intel.com] 发送时间: 2013年2月16日 11:10 收件人: user@hadoop.apache.org 主题: why my test result on dfs short

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
, did you enable security feature in your cluster? there'll be no obvious benefit be found if so. Regards, Liang ___ 发件人: Liu, Raymond [raymond@intel.com] 发送时间: 2013年2月16日 11:10 收件人: user@hadoop.apache.org 主题: why my test result on dfs short

答复: why my test result on dfs short circuit read is slower?

2013-02-15 Thread 谢良
security at all, please ignore my comments:) Regards, Liang 发件人: Liu, Raymond [raymond@intel.com] 发送时间: 2013年2月16日 11:40 收件人: user@hadoop.apache.org 主题: RE: why my test result on dfs short circuit read is slower? Hi Liang Did you mean set dfs.permissions

Re: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Harsh J
If you want HBase to leverage the shortcircuit, the DN config dfs.block.local-path-access.user should be set to the user running HBase (i.e. hbase, for example), and the hbase-site.xml should have dfs.client.read.shortcircuit defined in all its RegionServers. Doing this wrong could result in

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi Harsh Yes, I did set both of these. While not in hbase-site.xml but hdfs-site.xml. And I have double confirmed that local reads are performed, since there are no Error in datanode logs, and by watching lo network IO. If you want HBase to leverage the shortcircuit, the DN config

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for file

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
that read through datanode will be faster? Since it read data in bigger block size. Best Regards, Raymond Liu -Original Message- From: Liu, Raymond [mailto:raymond@intel.com] Sent: Saturday, February 16, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs