[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-11 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930040#comment-13930040
 ] 

Hadoop QA commented on HDFS-6080:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12633847/HDFS-6080.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-nfs:

  org.apache.hadoop.fs.TestHdfsNativeCodeLoader

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6369//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6369//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6369//console

This message is automatically generated.

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 |

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-11 Thread Brandon Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931033#comment-13931033
 ] 

Brandon Li commented on HDFS-6080:
--

[~ashahab], given your test result and hdfs file size is in MB size usually, I 
think we may want to keep 1MB as default. The user can always change it to a 
smaller size when needed. What do you think?

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-11 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931122#comment-13931122
 ] 

Hadoop QA commented on HDFS-6080:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12633847/HDFS-6080.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-nfs:

  org.apache.hadoop.fs.TestHdfsNativeCodeLoader

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6374//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6374//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6374//console

This message is automatically generated.

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 |

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-11 Thread Brandon Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931141#comment-13931141
 ] 

Brandon Li commented on HDFS-6080:
--

Yes. Let's update the doc too. If the user doesn't change the mount option and 
the nfs client uses 32kb or 64kb instead, the 1MB setting will not take effect 
so it hurts nothing.

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-11 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931307#comment-13931307
 ] 

Hadoop QA commented on HDFS-6080:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12634050/HDFS-6080.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-nfs:

  org.apache.hadoop.fs.TestHdfsNativeCodeLoader
  
org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6377//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/6377//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6377//console

This message is automatically generated.

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
>

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-13 Thread Brandon Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933590#comment-13933590
 ] 

Brandon Li commented on HDFS-6080:
--

+1

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-13 Thread Brandon Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934066#comment-13934066
 ] 

Brandon Li commented on HDFS-6080:
--

There are some format issues with the doc change. I've fixed them when 
committing the patch. Thank you, Abin, for the contribution! 

> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-13 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934069#comment-13934069
 ] 

Hudson commented on HDFS-6080:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5323 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5323/])
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax 
configurable. Contributed by Abin Shahab (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577319)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm


> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Fix For: 2.4.0
>
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch, 
> HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934872#comment-13934872
 ] 

Hudson commented on HDFS-6080:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/509/])
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax 
configurable. Contributed by Abin Shahab (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577319)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm


> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Fix For: 2.4.0
>
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch, 
> HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935026#comment-13935026
 ] 

Hudson commented on HDFS-6080:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1701 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1701/])
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax 
configurable. Contributed by Abin Shahab (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577319)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm


> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Fix For: 2.4.0
>
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch, 
> HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6080) Improve NFS gateway performance by making rtmax and wtmax configurable

2014-03-14 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935105#comment-13935105
 ] 

Hudson commented on HDFS-6080:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/])
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax 
configurable. Contributed by Abin Shahab (brandonli: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1577319)
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Constant.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm


> Improve NFS gateway performance by making rtmax and wtmax configurable
> --
>
> Key: HDFS-6080
> URL: https://issues.apache.org/jira/browse/HDFS-6080
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: nfs, performance
>Reporter: Abin Shahab
>Assignee: Abin Shahab
> Fix For: 2.4.0
>
> Attachments: HDFS-6080.patch, HDFS-6080.patch, HDFS-6080.patch, 
> HDFS-6080.patch
>
>
> Right now rtmax and wtmax are hardcoded in RpcProgramNFS3. These dictate the 
> maximum read and write capacity of the server. Therefore, these affect the 
> read and write performance.
> We ran performance tests with 1mb, 100mb, and 1GB files. We noticed 
> significant performance decline with the size increase when compared to fuse. 
> We realized that the issue was with the hardcoded rtmax size(64k). 
> When we increased the rtmax to 1MB, we got a 10x improvement in performance.
> NFS reads:
> +---++---+---+---++--+
> | File  | Size   | Run 1 | Run 2 | Run 3 
> | Average| Std. Dev.|
> | testFile100Mb | 104857600  | 23.131158137  | 19.24552955   | 19.793332866  
> | 20.72334018435 | 1.7172094782219731   |
> | testFile1Gb   | 1073741824 | 219.108776636 | 201.064032255 | 217.433909843 
> | 212.5355729113 | 8.14037175506561 |
> | testFile1Mb   | 1048576| 0.330546906   | 0.256391808   | 0.28730168
> | 0.291413464667 | 0.030412987573361663 |
> +---++---+---+---++--+
> Fuse reads:
> +---++-+--+--++---+
> | File  | Size   | Run 1   | Run 2| Run 3| 
> Average| Std. Dev. |
> | testFile100Mb | 104857600  | 2.394459443 | 2.695265191  | 2.50046517   | 
> 2.530063267997 | 0.12457410127142007   |
> | testFile1Gb   | 1073741824 | 25.03324924 | 24.155102554 | 24.901525525 | 
> 24.69662577297 | 0.386672412437576 |
> | testFile1Mb   | 1048576| 0.271615094 | 0.270835986  | 0.271796438  | 
> 0.271415839333 | 0.0004166483951065848 |
> +---++-+--+--++---+
> (NFS read after rtmax = 1MB)
> +---++--+-+--+-+-+
> | File  | Size   | Run 1| Run 2   | Run 3| 
> Average | Std. Dev.|
> | testFile100Mb | 104857600  | 3.655261869  | 3.438676067 | 3.557464787  | 
> 3.550467574336  | 0.0885591069882058   |
> | testFile1Gb   | 1073741824 | 34.663612417 | 37.32089122 | 37.997718857 | 
> 36.66074083135  | 1.4389615098060426   |
> | testFile1Mb   | 1048576| 0.115602858  | 0.106826253 | 0.125229976  | 
> 0.1158863623334 | 0.007515962395481867 |
> +---++--+-+--+-+-+



--
This message was sent by Atlassian JIRA
(v6.2#6252)