[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687792#comment-13687792
]
Steve Loughran commented on HDFS-4849:
--------------------------------------
I'm going to join in here because I've been deeply involved in FS semantics,
and I do have a passing knowledge of distributed systems.
bq. Create is idempotent iff {{create(path, NS) = create(path, create(path,
NS))}}
I must disagree.
First, there's an implicit: does some actual work. If create() always failed,
that would be idempotent, but not actually useful. Second, there's the whole
issue of what other state changes happen to the FS between the first attempt
and the second. It's not {{create(path, create(path, NS))}}, but {{create(path,
operation1(operation2(.. , operationN create(path, NS))}} for arbitrary
operations from other clients -which is why retry isn't sufficient.
Rather than try to write down some formal definitions of idempotent operations
on the state of a filesystem, I'll find some references and we can talk about
them. As it is, given we aren't at a consensus about the difference between
idempotent and repeated operations, trying to get this checked in is premature.
This isn't an API signature change - so it's not at the level of "apps need to
be recoded for it"; at the binary level then, it can go in later. It is
potentially changing the semantics of FS operations: provided we can be
confident that it really is going to be idempotent - to my happiness - then it
can go in later. I'm just nervous because we've dropped into a world where
proofs of correctness are going to be more important than tests, because we
need to show that concurrency issues during failure modes do not occur, and
small-scale tests aren't going to find that.
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch,
> idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
