Re: why my test result on dfs short circuit read is slower?

2013-02-18 Thread Michael Segel
On Feb 17, 2013, at 7:09 PM, Liu, Raymond raymond@intel.com wrote: io.file.buffer.size Drop this down to 64KB not 128KB. You have 16 cpu which really means 8 cores and 4 disks. Do you have Ganglia up and running? I'll wager that you'll see a lot of wait cpu cycles in both cases.

RE: why my test result on dfs short circuit read is slower?

2013-02-17 Thread Liu, Raymond
] Sent: Saturday, February 16, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs short circuit read is slower? Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short

RE: why my test result on dfs short circuit read is slower?

2013-02-17 Thread Liu, Raymond
, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs short circuit read is slower? Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi Liang Did you mean set dfs.permissions to false? Is that all I need to do to disable security feature? Cause It seems to me that without change dfs.block.local-path-access.user, dfs.permissions alone doesn't works. HBASE still fall back to go through datanode to read data. Hi Raymond,

Re: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Harsh J
If you want HBase to leverage the shortcircuit, the DN config dfs.block.local-path-access.user should be set to the user running HBase (i.e. hbase, for example), and the hbase-site.xml should have dfs.client.read.shortcircuit defined in all its RegionServers. Doing this wrong could result in

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi Harsh Yes, I did set both of these. While not in hbase-site.xml but hdfs-site.xml. And I have double confirmed that local reads are performed, since there are no Error in datanode logs, and by watching lo network IO. If you want HBase to leverage the shortcircuit, the DN config

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
Hi Arpit Gupta Yes, this way also confirms that short circuit read is enabled on my cluster. 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for file

RE: why my test result on dfs short circuit read is slower?

2013-02-15 Thread Liu, Raymond
that read through datanode will be faster? Since it read data in bigger block size. Best Regards, Raymond Liu -Original Message- From: Liu, Raymond [mailto:raymond@intel.com] Sent: Saturday, February 16, 2013 2:23 PM To: user@hadoop.apache.org Subject: RE: why my test result on dfs