[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834825#comment-13834825 ] Hudson commented on HDFS-5563: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1596 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1596/]) HDFS-5563. NFS gateway should commit the buffered data when read request comes after write to the same file. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1546233) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.2.1 > > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834813#comment-13834813 ] Hudson commented on HDFS-5563: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1622 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1622/]) HDFS-5563. NFS gateway should commit the buffered data when read request comes after write to the same file. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1546233) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.2.1 > > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834723#comment-13834723 ] Hudson commented on HDFS-5563: -- FAILURE: Integrated in Hadoop-Yarn-trunk #405 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/405/]) HDFS-5563. NFS gateway should commit the buffered data when read request comes after write to the same file. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1546233) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: 2.2.1 > > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834368#comment-13834368 ] Hudson commented on HDFS-5563: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4805 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4805/]) HDFS-5563. NFS gateway should commit the buffered data when read request comes after write to the same file. Contributed by Brandon Li (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1546233) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834363#comment-13834363 ] Brandon Li commented on HDFS-5563: -- Thank you, Jing. I've committed the patch. > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[
https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834319#comment-13834319
]
Hadoop QA commented on HDFS-5563:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616150/HDFS-5563.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5597//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5597//console
This message is automatically generated.
> NFS gateway should commit the buffered data when read request comes after
> write to the same file
>
>
> Key: HDFS-5563
> URL: https://issues.apache.org/jira/browse/HDFS-5563
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch,
> HDFS-5563.003.patch
>
>
> HDFS write is asynchronous and data may not be available to read immediately
> after write.
> One of the main reason is that DFSClient doesn't flush data to DN until its
> local buffer is full.
> To workaround this problem, when a read comes after write to the same file,
> NFS gateway should sync the data so the read request can get the latest
> content. The drawback is that, the frequent hsync() call can slow down data
> write.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834311#comment-13834311 ] Jing Zhao commented on HDFS-5563: - bq. In the future, we can consider use fix-sized cache blocks so that it's easy to find the cached data for read. Sounds good. +1 pending Jenkins. > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch, > HDFS-5563.003.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[
https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834046#comment-13834046
]
Hadoop QA commented on HDFS-5563:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616094/HDFS-5563.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5593//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5593//console
This message is automatically generated.
> NFS gateway should commit the buffered data when read request comes after
> write to the same file
>
>
> Key: HDFS-5563
> URL: https://issues.apache.org/jira/browse/HDFS-5563
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5563.001.patch, HDFS-5563.002.patch
>
>
> HDFS write is asynchronous and data may not be available to read immediately
> after write.
> One of the main reason is that DFSClient doesn't flush data to DN until its
> local buffer is full.
> To workaround this problem, when a read comes after write to the same file,
> NFS gateway should sync the data so the read request can get the latest
> content. The drawback is that, the frequent hsync() call can slow down data
> write.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[
https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832905#comment-13832905
]
Brandon Li commented on HDFS-5563:
--
Thanks for the review.
{quote}Instead of using fromRead as parameter, how about using a parameter
in the opposite way like "toCache"? Also please add javadoc for this new
parameter.{quote}
Javadoc added. "fromRead" make the meaning more obvious. For example, if the
commit is triggered, we should not update the stream access time.
{quote}It's better to use assertEquals(expected, actual) instead of
assertTrue(expected == actual value) in the unit test.{quote}
Done.
{quote} A possible optimization here may be to directly return the local
buffered data for the read request without calling hsync. This may be addressed
in future jiras.
{quote}
Current cached data in pending write hashmap is not searchable. In the future,
we can consider use fix-sized cache blocks so that it's easy to find the cached
data for read.
> NFS gateway should commit the buffered data when read request comes after
> write to the same file
>
>
> Key: HDFS-5563
> URL: https://issues.apache.org/jira/browse/HDFS-5563
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5563.001.patch
>
>
> HDFS write is asynchronous and data may not be available to read immediately
> after write.
> One of the main reason is that DFSClient doesn't flush data to DN until its
> local buffer is full.
> To workaround this problem, when a read comes after write to the same file,
> NFS gateway should sync the data so the read request can get the latest
> content. The drawback is that, the frequent hsync() call can slow down data
> write.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[
https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832284#comment-13832284
]
Hadoop QA commented on HDFS-5563:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12615764/HDFS-5563.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5571//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5571//console
This message is automatically generated.
> NFS gateway should commit the buffered data when read request comes after
> write to the same file
>
>
> Key: HDFS-5563
> URL: https://issues.apache.org/jira/browse/HDFS-5563
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5563.001.patch
>
>
> HDFS write is asynchronous and data may not be available to read immediately
> after write.
> One of the main reason is that DFSClient doesn't flush data to DN until its
> local buffer is full.
> To workaround this problem, when a read comes after write to the same file,
> NFS gateway should sync the data so the read request can get the latest
> content. The drawback is that, the frequent hsync() call can slow down data
> write.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5563) NFS gateway should commit the buffered data when read request comes after write to the same file
[ https://issues.apache.org/jira/browse/HDFS-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832212#comment-13832212 ] Jing Zhao commented on HDFS-5563: - The patch looks good overall. Some minor: # Instead of using fromRead as parameter, how about using a parameter in the opposite way like "toCache"? Also please add javadoc for this new parameter. # It's better to use assertEquals(expected, actual) instead of assertTrue(expected == actual value) in the unit test. # A possible optimization here may be to directly return the local buffered data for the read request without calling hsync. This may be addressed in future jiras. > NFS gateway should commit the buffered data when read request comes after > write to the same file > > > Key: HDFS-5563 > URL: https://issues.apache.org/jira/browse/HDFS-5563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5563.001.patch > > > HDFS write is asynchronous and data may not be available to read immediately > after write. > One of the main reason is that DFSClient doesn't flush data to DN until its > local buffer is full. > To workaround this problem, when a read comes after write to the same file, > NFS gateway should sync the data so the read request can get the latest > content. The drawback is that, the frequent hsync() call can slow down data > write. -- This message was sent by Atlassian JIRA (v6.1#6144)
