On Feb 17, 2013, at 7:09 PM, Liu, Raymond raymond@intel.com wrote:
io.file.buffer.size
Drop this down to 64KB not 128KB.
You have 16 cpu which really means 8 cores and 4 disks.
Do you have Ganglia up and running?
I'll wager that you'll see a lot of wait cpu cycles in both cases.
]
Sent: Saturday, February 16, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs short circuit read is slower?
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my
cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short
, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs short circuit read is slower?
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my
cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
Hi Liang
Did you mean set dfs.permissions to false?
Is that all I need to do to disable security feature? Cause It seems to me that
without change dfs.block.local-path-access.user, dfs.permissions alone doesn't
works. HBASE still fall back to go through datanode to read data.
Hi Raymond,
If you want HBase to leverage the shortcircuit, the DN config
dfs.block.local-path-access.user should be set to the user running
HBase (i.e. hbase, for example), and the hbase-site.xml should have
dfs.client.read.shortcircuit defined in all its RegionServers. Doing
this wrong could result in
Hi Harsh
Yes, I did set both of these. While not in hbase-site.xml but hdfs-site.xml.
And I have double confirmed that local reads are performed, since there are no
Error in datanode logs, and by watching lo network IO.
If you want HBase to leverage the shortcircuit, the DN config
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for file
that read through datanode will be
faster? Since it read data in bigger block size.
Best Regards,
Raymond Liu
-Original Message-
From: Liu, Raymond [mailto:raymond@intel.com]
Sent: Saturday, February 16, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs