Alright, I think in my sequence read scenario, it is possible that shortcircuit read is actually slower than read through datanode.
For, when read through datanode, FS read operation is done by datanode daemon, while data processing is done by client. Thus when client is processing the data, data node could read data at the same time and write it to local socket for client to read. It take very few time for client to read from local socket. While when client read native FS directly, all the job is done by client. It will be blocked for more time to read data from native FS for process than from local socket. Overall, when CPU is not bound, and data node always prepare further data for client(due to sequence read scenario), the result is that shortcircuit read is slower though it cost less CPU load. The CPU/idle/IOWait load seems also justify my guess = For a scan only job: = Read through datanode: CPU : 30-35% / IOWait : ~50% / 300 second Shortcircuit read: CPU : 25-30% / IOWait : ~40% / 330 second Short circuit read is 10% slower = For a job do more calculation = Read through datanode: CPU : 80-90% / IOWait : ~5-10% / 190 seconds Shortcircuit read: CPU : ~90% / IOWait : ~2-3% / 160 second Short circuit read is 15% faster. So, short circuit read is not always faster, especially when CPU is not bound and read by sequence, it will be slower. This is the best explain I can get now. Any thoughts? Raymond > > I have try to tune io.file.buffer.size to 128K instead of 4K ShortCircuit read > performance is still worse than read through datanode. > > I am start to wondering, does shortcircuit read really help under hadoop 1.1.1 > version? > I google to find a few people mention they got 2x gain or so upon CDH etc. > But I > really can't find out what else I can do to make it even just catch up normal > read path.... > > > > > It seems to me that, with short circuit read enabled, the > > BlockReaderLocal read data in 512/4096 bytes unit(checksum check > > enabled/skiped) > > > > While when It go through datanode, the BlockSender.sendChunks will > > read and sent data in 64K bytes units? > > > > Is that true? And if so, won't it explain that read through datanode > > will be faster? Since it read data in bigger block size. > > > > Best Regards, > > Raymond Liu > > > > > > > -----Original Message----- > > > From: Liu, Raymond [mailto:raymond....@intel.com] > > > Sent: Saturday, February 16, 2013 2:23 PM > > > To: user@hadoop.apache.org > > > Subject: RE: why my test result on dfs short circuit read is slower? > > > > > > Hi Arpit Gupta > > > > > > Yes, this way also confirms that short circuit read is enabled on my > cluster. > > > > > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: Short circuit read is true > > > > > > 13/02/16 14:07:34 DEBUG hdfs.DFSClient: New BlockReaderLocal for > > > file > > > > > > /mnt/DP_disk4/raymond/hdfs/data/current/subdir63/blk_-2736548898990727 > > > 638 of size 134217728 startOffset 0 length 134217728 short circuit > > > checksum false > > > > > > So , any possibility that other setting might impact short circuit > > > read to has worse performance than read through datanode? > > > > > > Raymond > > > > > > > > > > > > >Another way to check if short circuit read is configured correctly. > > > > > > >As the user who is configured for short circuit read issue the > > > >following > > > command on a node where you expect the data to be read locally. > > > > > > >export HADOOP_ROOT_LOGGER=debug,console; hadoop dfs -cat > > > >/path/to/file_on_hdfs > > > > > > >On the console you should see something like "hdfs.DFSClient: New > > > BlockReaderLocal for file...." > > > > > > >This would confirm that short circuit read is happening. > > > > > > -- > > > >Arpit Gupta > > > >Hortonworks Inc. > > > >http://hortonworks.com/ > > > > > > On Feb 15, 2013, at 9:53 PM, "Liu, Raymond" <raymond....@intel.com> > > wrote: > > > > > > > > > Hi Harsh > > > > > > Yes, I did set both of these. While not in hbase-site.xml but > > > hdfs-site.xml. > > > > > > And I have double confirmed that local reads are performed, since > > > there are no Error in datanode logs, and by watching lo network IO. > > > > > > > > > > > > If you want HBase to leverage the shortcircuit, the DN config > > > "dfs.block.local-path-access.user" should be set to the user running > > > HBase > > (i.e. > > > hbase, for example), and the hbase-site.xml should have > > > "dfs.client.read.shortcircuit" defined in all its RegionServers. > > > Doing this wrong could result in performance penalty and some > > > warn-logging, as local reads will be attempted but will begin to fail. > > > > > > On Sat, Feb 16, 2013 at 8:40 AM, Liu, Raymond > > > <raymond....@intel.com> > > > wrote: > > > > > > Hi > > > > > > I tried to use short circuit read to improve my hbase cluster > > > MR scan performance. > > > > > > > > > I have the following setting in hdfs-site.xml > > > > > > dfs.client.read.shortcircuit set to true > > > dfs.block.local-path-access.user set to MR job runner. > > > > > > The cluster is 1+4 node and each data node have 16cpu/4HDD, > > > with all hbase table major compact thus all data is local. > > > > > > I have hoped that the short circuit read will improve the > > > performance. > > > > > > > > > While the test result is that with short circuit read > > > enabled, the performance actually dropped 10-15%. Say scan a 50G > > > table cost around 100s instead of 90s. > > > > > > > > > My hadoop version is 1.1.1, any idea on this? Thx! > > > > > > Best Regards, > > > Raymond Liu > > > > > > > > > > > > > > > -- > > > Harsh J
