[
https://issues.apache.org/jira/browse/FLUME-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463443#comment-13463443
]
Mike Percy commented on FLUME-1573:
-----------------------------------
Many people run a cluster of collector agents, so they have multiple agents
writing to HDFS. Say you have a config file with an HDFS sink that looks like
this:
{noformat}
agent.sinks.sink-0.type = hdfs
agent.sinks.sink-0.channel = ch-0
agent.sinks.sink-0.hdfs.batchSize = 1000
agent.sinks.sink-0.hdfs.path = /user/flume/events
agent.sinks.sink-1.hdfs.filePrefix = events-0
agent.sinks.sink-0.hdfs.fileType = DataStream
agent.sinks.sink-0.hdfs.rollCount = 0
agent.sinks.sink-0.hdfs.rollSize = 0
agent.sinks.sink-0.hdfs.rollInterval = 300
{noformat}
If you have multiple machines running with this config, they will overwrite
each other's HDFS files. A static counter will not help with that case. My
argument is that people need to be aware of this danger, and so preventing them
from hitting this issue in a single-agent case makes it more likely that they
will not find out about this problem until much later.
Back to the matter at hand. The correct way to add a 2nd HDFS sink in the same
agent is to give each sink a unique filePrefix (or path). In the case of
running multiple collector machines using the same config, the simple approach
is to add a HOST interceptor to the source, and then reference the %{host}
header in the {{hdfs.path}} or {{hdfs.filePrefix}} of each sink.
> Duplicated HDFS file name when multiple SinkRunner was existing
> ---------------------------------------------------------------
>
> Key: FLUME-1573
> URL: https://issues.apache.org/jira/browse/FLUME-1573
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Fix For: v1.3.0
>
> Attachments: FLUME-1573.patch
>
>
> Multiple HDFS Sinks to write events into storage. Timeout exception is always
> happening:
> {code:xml}
> 11 Sep 2012 07:04:53,478 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:442) - HDFS IO error
> java.io.IOException: Callable timed out after 10000 ms
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:342)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.append(HDFSEventSink.java:713)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:412)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.concurrent.TimeoutException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:335)
> ... 5 more
> {code}
> I doubted that there might be happened HDFS timeout or slowly response. As
> expected, I found the duplicated creation exception with same with at HDFS.
> Also, Flume recorded same case for duplicated file name.
> {code:xml}
> 13 Sep 2012 02:09:35,432 INFO [hdfs-hdfsSink-3-call-runner-7]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:189) - Creating
> /FLUME/dt=2012-09-13/02-host.1347501924111.tmp
> 13 Sep 2012 02:09:36,425 INFO [hdfs-hdfsSink-4-call-runner-8]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:189) - Creating
> /FLUME/dt=2012-09-13/02-host.1347501924111.tmp
> {code}
> Different threads were going to create same file without time conflict.
> I found the root cause might be wrong usage the AtomicLong property named
> 'fileExtensionCounter' at BucketWriter. Different threads should own same
> counter by protected with CAS, not multiple private property in each thread.
> It's useless to avoid conflict of HDFS path
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
