Jifeng Yin created HDFS-6365: -------------------------------- Summary: slow performance when reading big file (~10G) Key: HDFS-6365 URL: https://issues.apache.org/jira/browse/HDFS-6365 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.4.0 Reporter: Jifeng Yin
Mount options: {code}rw,noatime,nodiratime,vers=3,rsize=65536,wsize=65536,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys{code} Read a big file (~10G) , {code} time dd if=/file/from/hdfs bs=4M | pv | dd of=/dev/null {code} Results: First time: 1. ~160MB/s at first, without cached memory increased on the nfs server 2. ~40MB/s with cached memory increased ( cached memory is always kept). logs show a ton of the following records: {quote} 2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_END______786375175 2014-05-09 15:36:13,819 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: READ_RPC_CALL_START____803152391 2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: NFS READ fileId: 17258 offset: 3243958272 count: 65536 2014-05-09 15:36:13,820 DEBUG org.apache.hadoop.hdfs.nfs.nfs3.WriteManager: No opened stream for fileId:17258 commitOffset=3244023808. Return success in this case. {quote} Second time: the same as first time 2 phrase. -- This message was sent by Atlassian JIRA (v6.2#6252)