[
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425739#comment-16425739
]
Steve Loughran commented on HADOOP-15361:
-----------------------------------------
seems good, though there's still some funnies about Windows ::MoveFile(), which
will switch to a copy if the dest is on a different volume.
* revert the import .* changes; maybe look at your IDE settings there to keep
that down.
* in {{TestRawLocalFileSystemContract}} some new tests. Are they common to all
filesystems? If so I'd like them in {{AbstractContractRenameTest}}, though that
will force you to test against the object stores too, I'm afraid.
The compatibility is the troublespot here. How does it relate to what we have
in filesystem.md?
The normative behaviour we want is that of HDFS. If you put the new tests in
{{AbstractContractRenameTest}}, the HDFS subclass {{TestHDFSContractRename}}
must pass them. If it doesn't, that's a problem in the tests or the changed
behaviour.
I'm less worried about backwards compatibility with RawLocal than I am with
consistency with HDFS, because that's what we try to do everywhere: make things
work like that, except for the bits that don't.
Bear in mind that posix rename() is slightly different from HDFS rename() w.r.t
empty directories. That is believed to be an accidental mistake in the HDFS
behaviour that we are all stuck with. So if the outcome of the contract tests
are different, well, we will have to consider a new option to enable/disable in
the test contract.
Now, what happens on windows?
> RawLocalFileSystem should use Java nio framework for rename
> -----------------------------------------------------------
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Andras Bokor
> Assignee: Andras Bokor
> Priority: Major
> Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames.
> The fallback logic is a copy-on-fail logic so when rename fails it copies the
> source then delete it.
> An additional fallback logic was needed for Windows to provide POSIX rename
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io
> overrides the destination only if the source is also a directory but
> handleEmptyDstDirectoryOnWindows method checks only the destination. That
> means rename allows to override a directory with a file on Windows but not on
> Unix.
> File#renameTo and Files#move are not 100% compatible:
> If the source is a directory and the destination is an empty directory
> File#renameTo overrides the source but Files#move is does not. We have to use
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even
> if the source or the destination is a file. So to make them compatible we
> have to check that the either the source or the destination is a directory
> before we add the copy option.
> I think the correct strategy is
> * Where the contract test passed so far it should pass after this
> * Where the contract test failed because of Java specific think and not
> because of the fallback logic we should keep the original behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
