[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686381#comment-13686381
]
Suresh Srinivas commented on HDFS-4849:
---------------------------------------
bq. If our goal is to build a file system that is better than NFS which is,
after all, twenty year old technology, we should consider making all operations
that modify the file system metadata into idempotent operations.
I should have made this point clear. If you are supporting this change, then I
want to point out, this change is not making create or append idempotent. It is
merely handling a retry scenario. It allows from the same client a retry
create. Truly idempotent operation should allow create for all clients and the
result of operation should be the same.
bq. As I understand it, by default the FileSystem instance in the client is
shared by all threads in the JVM. Doesn't this imply that the program should
either be single threaded, or it should use its own synchronization of
operations between threads? If it is using its own synchronization, then the
above problem will not happen.
The issue is not about synchronization. When the file system is created, during
initialization client name is set. This same client name is used as identity of
client for all the subsequent communication to the namenode. This identity is
used in the current patch for allowing retry. If two threads use the same file
system instance, even though the client side is correctly synchronized, they
end up using the same identity.
bq. If the operation is defined as being idempotent, then having two entries in
the edits log is harmless, is it not?
I am talking about auditlog.
{quote}
HDFS has an option to turn FileSystem caching off, which is another way to
avoid multi-threaded clientName collisions.
Multiple threads using the same cached FileSystem has an issue even now without
retries.
Suppose one creates a file and then spawns several threads that start creating
blocks simultaneously on the same file. The same problem, right?
{quote}
This is not just a problem with caching. Multiple threads can use a single
instance of a file system.
As regards creating blocks simultaneously, it is prevented precisely because
multiple creates are not allowed today and hence the application does not even
get to allocating blocks. That is exactly the point I am raising.
bq. Is there a requirement for audit log to print the create file only once?
Yes. If you log it multiple times, it is a change in behavior. Some could today
use audit log to construct the sequence of events happened. Currently it has a
single create, followed by modifies/deletes. Two creates without being a delete
in the middle is bug today. Suddenly with this such sequence of logs are
possible. For example external auditlog listeners could be configured in 2.x,
which could be inserting a row to a DB. They could fail, with the duplicate
entries.
If auditlog duplicate entires are okay, should editlog have two entries?
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch,
> idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
