[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685998#comment-13685998
]
Suresh Srinivas commented on HDFS-4849:
---------------------------------------
bq. I elaborated on multithreaded clients in the same comment as Todd raised
this question. See second part.
Assuming that it is the following point.
bq. If you go through FileSystem API that should not happen, because
DFSClient.clientName includes the thread name. Yes, I can write a test which
directly makes RPC calls to NN and fakes them with the same clientName for
different threads. But this is not public HDFS APIs, and there is many other
ways to abuse the system. Should we care?
Client name includes the name of the thread where it was created in. However,
one could use the same FileSystem instance and hence same DFSClient from
multiple threads.
bq. To summarize: current HDFS takes client names from applications. MR
guarantees the uniqueness. The other applications should too.
Applications cannot set the client name. Even if they do, multiple threads can
reuse the same client name. I do not think one can assume that applications are
doing the right thing and setting a unique name. Also in case of YARN, where
multiple application masters could be running, not sure if there is anyway to
guarantee this. Yarn or MR folks, can you comment?
bq. Well if there is a bug we should fix it.
Isn't two creates logged in an audit log instead of a single create not a bug?
bq. I proposed to reuse the jira for create and append in this comment. Created
HDFS-4872 for the delete work with this issue linked to it. I don't mind
reorganizing jiras as you wish.
Creating a separate jira may be a good idea, given all the discussion this jira
has, including delete. Also currently the description still says create, append
and delete, where as the summary talks only about create and append.
My thoughts are, instead of trying to make these operations as idempotent, the
right thing to do is to make them retry safe. This could be done by:
# Retry cache as suggested previously
# Change RPC to flag a request as retried, allow retry for a given call ID in a
small window of time, (which means now we need to record the RPC call ID that
created or appended to a file + client name). I have not thought through this
option. This still may not be the correct solution.
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
