[ https://issues.apache.org/jira/browse/FLUME-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388701#comment-15388701 ]
Jeff Field commented on FLUME-2458: ----------------------------------- Unfortunately, distcp copies dotfiles, unless you regex exclude them but the regex exclude carries over even when a file is renamed apparently, so distcp just will never copy the file (until you kill the snapshots and have it do a non-snapshot run). That is a good solutions for hiding things from things that aren't distcp though, yeah. > Separate hdfs tmp directory for flume hdfs sink > ----------------------------------------------- > > Key: FLUME-2458 > URL: https://issues.apache.org/jira/browse/FLUME-2458 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources > Affects Versions: v1.5.0.1 > Reporter: Sverre Bakke > Assignee: Neerja Khattar > Priority: Minor > Attachments: FLUME-2458.patch, patch-2458.txt > > > The current HDFS sink will write temporary files to the same directory as the > final file will be stored. This is a problem for several reasons: > 1) File moving > When mapreduce fetches a list of files to be processed and then processes > files that are then gone (i.e. are moved from .tmp to whatever final name it > is suppose to have), then the mapreduce job will crash. > 2) File type > When mapreduce decides how to process files, then it looks at files > extension. If using compressed files, then it will decompress it for you. If > the file has a .tmp file extension (in the same folder) then it will treat a > compressed file as an uncompressed files, thus breaking the results of the > mapreduce job. > I propose that the sink gets an optional tmp path for storing these files to > avoid these issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)