[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706397#comment-13706397
]
Colin Patrick McCabe commented on HDFS-4849:
--------------------------------------------
Hi Konstantin,
I understand your frustration with the slow pace of progress on this JIRA.
Believe me. But this is different than some of the other JIRAs we've done.
Usually, there is a configuration option that can turn off the new thing if
there is a problem (as with HDFS-347) or people can continue using the old
system (as with QuorumJournalManager). People can't exactly turn off file
creation-- it's a core part of the filesystem. So we want to make sure that we
get everything right the first time, even if that means some pain up front.
I read through the comments here, right from the start, as well as the patch.
It seems like the main issues raised were:
1. semantics-- some people are uncomfortable using the annotation "Idempotent"
2. problems with multithreaded behavior
Point #1 seems like something we can resolve without too much difficulty. We
can change the '@Idempotent' annotation to '@SkipRetryCache' or something like
that. I see that you've already renamed the JIRA to "something like that" :)
Point #2 was originally brought up by [~tlipcon]. He wrote:
bq. The idempotent create is also potentially problematic with multi-threaded
clients. A given client may have multiple threads race to create the same file,
and those threads would share the same client name (and hence lease).
If I understand correctly, Konstantin's response was that users could use
{{FileSystem#newInstance}} to give each thread a separate DFSClient (and hence
their own client names). However, that doesn't help the people who are relying
on the current behavior.
In HDFS-4942, Suresh proposes "adding a UUID as the request ID in the RPC
client." It seems like that might be helpful in resolving #2. Does it make
sense to collaborate on this?
> Enable retries for create and append operations.
> ------------------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate-branch2.patch, idempotentCreate.patch,
> idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
>
>
> create, append and delete operations can be made retriable. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
