[
https://issues.apache.org/jira/browse/FLUME-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371961#comment-15371961
]
Mike Percy commented on FLUME-2458:
-----------------------------------
[~jfield], maybe I'm missing something but why wouldn't this work?
{code}
tier1.sinks.bi_prod_sink.hdfs.path =
/DW/App/%{routing_key}/%{x-proto-message-type}%{fileName}%{proto}%{message_id}/%Y%m%d/
tier1.sinks.bi_prod_sink.hdfs.filePrefix = $H-
tier1.sinks.bi_prod_sink.hdfs.fileSuffix = .txt
tier1.sinks.bi_prod_sink.hdfs.inUsePrefix = .
{code}
> Separate hdfs tmp directory for flume hdfs sink
> -----------------------------------------------
>
> Key: FLUME-2458
> URL: https://issues.apache.org/jira/browse/FLUME-2458
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.5.0.1
> Reporter: Sverre Bakke
> Assignee: Neerja Khattar
> Priority: Minor
> Attachments: FLUME-2458.patch, patch-2458.txt
>
>
> The current HDFS sink will write temporary files to the same directory as the
> final file will be stored. This is a problem for several reasons:
> 1) File moving
> When mapreduce fetches a list of files to be processed and then processes
> files that are then gone (i.e. are moved from .tmp to whatever final name it
> is suppose to have), then the mapreduce job will crash.
> 2) File type
> When mapreduce decides how to process files, then it looks at files
> extension. If using compressed files, then it will decompress it for you. If
> the file has a .tmp file extension (in the same folder) then it will treat a
> compressed file as an uncompressed files, thus breaking the results of the
> mapreduce job.
> I propose that the sink gets an optional tmp path for storing these files to
> avoid these issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
