[
https://issues.apache.org/jira/browse/HDFS-9763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174219#comment-15174219
]
Jitendra Nath Pandey commented on HDFS-9763:
--------------------------------------------
I don't think this API will address TOCTOU because hive will need multiple
merge calls anyway for a given query, to reach final stage. In the absence of
transactional support in the file system, as suggested by Haohui, it is
difficult to support such semantics at HDFS level.
However, I do see the need of hive to avoid hundreds or thousands of
individual rename operations. This also seems to be a pretty general case for
many map-reduce based data ingest jobs that write data all over the place and
finally need to consolidate in a final directory, after all the related jobs
are successful.
IMO, the requirement here is of a 'rename' operation that merges the
directories instead of overwriting, or throwing already-exists-exception.
However, it can be argued that its better to add a new 'merge' API, instead of
overloading the 'rename'.
I see that there are two main design points here that need to be carefully
thought through and agreed upon
1) How do we resolve conflicts in file names
I think the proposal handles it elegantly by specifying a policy. In fact I
would love to change our 'rename' to support different policies that also
provides merge capability, but I can accept a separate API for compatibility
and simplicity.
2) The O(N) problem.
It is not so bad because its not recursive. There is no scope of recursion
here.
Still if a directory has a lot of files, an iterative approach is feasible
here, because the source directory will get smaller after every iteration. We
do have precedence of iteration for example 'listStatus'. This will avoid the
complexity in the NN to release the lock in between.
> Add merge api
> -------------
>
> Key: HDFS-9763
> URL: https://issues.apache.org/jira/browse/HDFS-9763
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Reporter: Ashutosh Chauhan
> Assignee: Xiaobing Zhou
> Attachments: HDFS_Merge_API_Proposal.pdf
>
>
> It will be good to add merge(Path dir1, Path dir2, ... ) api to HDFS.
> Semantics will be to move all files under dir1 to dir2 and doing a rename of
> files in case of collisions.
> In absence of this api, Hive[1] has to check for collision for each file and
> then come up unique name and try again and so on. This is inefficient in
> multiple ways:
> 1) It generates huge number of calls on NN (atleast 2*number of source files
> in dir1)
> 2) It suffers from TOCTOU[2] bug for client picked up name in case of
> collision.
> 3) Whole operation is not atomic.
> A merge api outlined as above will be immensely useful for Hive and
> potentially to other HDFS users.
> [1]
> https://github.com/apache/hive/blob/release-2.0.0-rc1/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2576
> [2] https://en.wikipedia.org/wiki/Time_of_check_to_time_of_use
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
