[
https://issues.apache.org/jira/browse/MAPREDUCE-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13713743#comment-13713743
]
Robert Joseph Evans commented on MAPREDUCE-5247:
------------------------------------------------
Why are you running a Map/Reduce job with input from a directory that has not
finished being copied? MR was not designed to run on data that is changing
underneath it. When the job is done how do you know which of the input files
were actually used to produce the output? This issue existed prior to 2.0 but
was even worse without the ._COPYING_ suffix. In those cases the files were
opened in place and data started to be copied into them. You may have only
even gotten a part of the file in your MR job, not all of it. The file could
have disappeared out from under the MR job if an error occurred.
This is not behavior that I want to make a common park of Map/Reduce. If you
want to do this and you know the risks then you can filter ._COPYING_ files out
of your list of input files to the MR job. But I don't want the framework to
do it automatically for everyone.
> FileInputFormat should filter files with '._COPYING_' sufix
> -----------------------------------------------------------
>
> Key: MAPREDUCE-5247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5247
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Stan Rosenberg
>
> FsShell copy/put creates staging files with '._COPYING_' suffix. These files
> should be considered hidden by FileInputFormat. (A simple fix is to add the
> following conjunct to the existing hiddenFilter:
> {code}
> !name.endsWith("._COPYING_")
> {code}
> After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data
> loader which uses 'hadoop fs -put' to load data into hourly partitions. We
> also have intra-hourly jobs which are scheduled to execute several times per
> hour using the same hourly partition as input. Thus, as the new data is
> continuously loaded, these staging files (i.e., ._COPYING_) are breaking our
> jobs (since when copy/put completes staging files are moved).
> As a workaround, we've defined a custom input path filter and loaded it with
> "mapred.input.pathFilter.class".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
