[
https://issues.apache.org/jira/browse/HDFS-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980876#comment-13980876
]
Steve Loughran commented on HDFS-6262:
--------------------------------------
Suresh, thanks for the link to HADOOP-6240 -I hadn't seen that. But: *every
other filesystem* considers renaming a file that doesn't exist to be an error.
Do we have any examples where failing to fault on renaming a nonexistent file
is NOT an error to flag up?
Looking at the hadoop production source
* {{org.apache.hadoop.fs.shell.MoveCommands}} says "we have no way to know the
actual error..." and throws a {{PathIOException}}
* {{org.apache.hadoop.fs.shell.CommandWithDestination}} says "too bad we don't
know why it failed" and does the same
* {{org.apache.hadoop.io.MapFile}} raises an IOException
* {{org.apache.hadoop.tools.mapred.CopyCommitter}} raises an IOE, as does
{{org.apache.hadoop.tools.mapred.RetriableFileCopyCommand}}
Similar behaviour for:
{code}
LocalContainerLauncher, DistCpV1
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
org.apache.hadoop.mapreduce.v2.hs.HistoryServerFileSystemStateStoreService,
...
{code}
and those that blindly assume that rename's return value doesn't need checking
{code}
JobHistoryEventHandler
TaskLog (on localFS though)
org.apache.hadoop.mapreduce.task.reduce.OnDiskMapOutput
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
{code}
In fact. the only bit of code I can see that converts the false return code to
a warning is {{org.apache.hadoop.tools.mapred.lib.DynamicInputChunk}}
To summarise, in the Hadoop production code, in all but one case the handling
of a false return code takes two forms
# triggers the throwing of a "that failed but we don't know why" {{IOException}}
# is blissfully ignorant that the operation has failed, and has so far been
lucky in avoiding concurrency problems with their source being renamed while
they weren't looking.
All of these uses benefit from having rename consistently throw a
FileNotFoundException if the source file isn't there
> HDFS doesn't raise FileNotFoundException if the source of a rename() is
> missing
> -------------------------------------------------------------------------------
>
> Key: HDFS-6262
> URL: https://issues.apache.org/jira/browse/HDFS-6262
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.4.0
> Reporter: Steve Loughran
> Assignee: Akira AJISAKA
> Attachments: HDFS-6262.2.patch, HDFS-6262.patch
>
>
> HDFS's {{rename(src, dest)}} returns false if src does not exist -all the
> other filesystems raise {{FileNotFoundException}}
> This behaviour is defined in {{FSDirectory.unprotectedRenameTo()}} -the
> attempt is logged, but the operation then just returns false.
> I propose changing the behaviour of {{DistributedFileSystem}} to be the same
> as that of the others -and of {{FileContext}}, which does reject renames with
> nonexistent sources
--
This message was sent by Atlassian JIRA
(v6.2#6252)
