On Feb 17, 2013, at 7:09 PM, Liu, Raymond raymond@intel.com wrote:
io.file.buffer.size
Drop this down to 64KB not 128KB.
You have 16 cpu which really means 8 cores and 4 disks.
Do you have Ganglia up and running?
I'll wager that you'll see a lot of wait cpu cycles in both cases.
]
Sent: Saturday, February 16, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs short circuit read is slower?
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my
cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short
, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs short circuit read is slower?
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my
cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
[raymond@intel.com]
发送时间: 2013年2月18日 14:04
收件人: user@hadoop.apache.org
主题: RE: why my test result on dfs short circuit read is slower?
Alright, I think in my sequence read scenario, it is possible that shortcircuit
read is actually slower than read through datanode.
For, when read through datanode
Hi
I tried to use short circuit read to improve my hbase cluster MR scan
performance.
I have the following setting in hdfs-site.xml
dfs.client.read.shortcircuit set to true
dfs.block.local-path-access.user set to MR job runner.
The cluster is 1+4 node
Hi Raymond,
did you enable security feature in your cluster? there'll be no obvious
benefit be found if so.
Regards,
Liang
___
发件人: Liu, Raymond [raymond@intel.com]
发送时间: 2013年2月16日 11:10
收件人: user@hadoop.apache.org
主题: why my test result on dfs short
,
did you enable security feature in your cluster? there'll be no obvious
benefit
be found if so.
Regards,
Liang
___
发件人: Liu, Raymond [raymond@intel.com]
发送时间: 2013年2月16日 11:10
收件人: user@hadoop.apache.org
主题: why my test result on dfs short
security at all, please ignore my comments:)
Regards,
Liang
发件人: Liu, Raymond [raymond@intel.com]
发送时间: 2013年2月16日 11:40
收件人: user@hadoop.apache.org
主题: RE: why my test result on dfs short circuit read is slower?
Hi Liang
Did you mean set dfs.permissions
If you want HBase to leverage the shortcircuit, the DN config
dfs.block.local-path-access.user should be set to the user running
HBase (i.e. hbase, for example), and the hbase-site.xml should have
dfs.client.read.shortcircuit defined in all its RegionServers. Doing
this wrong could result in
Hi Harsh
Yes, I did set both of these. While not in hbase-site.xml but hdfs-site.xml.
And I have double confirmed that local reads are performed, since there are no
Error in datanode logs, and by watching lo network IO.
If you want HBase to leverage the shortcircuit, the DN config
Hi Arpit Gupta
Yes, this way also confirms that short circuit read is enabled on my cluster.
13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true
13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for file
that read through datanode will be
faster? Since it read data in bigger block size.
Best Regards,
Raymond Liu
-Original Message-
From: Liu, Raymond [mailto:raymond@intel.com]
Sent: Saturday, February 16, 2013 2:23 PM
To: user@hadoop.apache.org
Subject: RE: why my test result on dfs
12 matches
Mail list logo