[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13641855#comment-13641855
]
Brock Noland commented on HDFS-4750:
------------------------------------
Hi Brandon,
Thank you for the quick response!
bq. "10 milliseconds" is the time from the reference paper. In the initial
implementation, we used 10 seconds just to be on the safe side.
What happens if the 10 seconds expires and the prerequisite write has not been
received? The biggest issue I had when moving the proxy from basically working
to handling multiple heavy use clients was memory consumption while waiting for
pre-requisite writes. I eventually had to write pending writes to a file. This
is documented in this issue https://github.com/brockn/hdfs-nfs-proxy/issues/7
bq. Regarding small file append, it starts from the correct offset in the tests
I observed. For example, I tried "echo abcd >> /mnt_test/file_with_5bytes", the
write request starts with offset 5. With the initial file loading tests with
Linux/Mac clients, so far we haven't encountered the problem you mentioned.
Interesting, what version of linux have tried? I believe I was using RHEL 5.X.
bq. For the second question, as long as the second user uses NFS gateway to
read the closed file, the second user should be able to get the data buffered
in NFS gateway. For the opened files, NFS gateway also saves their latest file
size. When it serves getattr request, it gets file attributes from HDFS and
then update the file length based on the cached length.
Cool, my question was more around how we are going to make our users aware of
this limitation. I could imagine many users believing once they have closed a
file via NFS that exact file will be available via one the other APIs. We'll
need to make this limitation blatantly obvious to users otherwise it will
likely become a support headache.
Additionally, is there anything the user can do to force the writes? i.e. If
the user has control over the program, could they do a fsync(fd) to force the
flush?
Cheers,
Brock
> Support NFSv3 interface to HDFS
> -------------------------------
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.0.0
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
