Hello Shara,
Unfortunately tailing is actually really complicated because of what can
happen when files get moved around and such, and as a result we don't
plan to include a taildir source like there was in OG.
If you use "tail -0F" for your command you will not get any repeated
lines, and will get around file renames.
If you want more specific functionality I think the best approach is to
implement a program in python or some other language that allows access
to OS specific details(such as inodes in linux) and that you send the
data to flume via avro.
On 06/20/2012 04:36 PM, Shara Shi wrote:
Hi Mohammad Tariq ,
If the agent restart , tail command would be execute again that means
duplicated data are transferred to sink.
How do I deal with this situation?
Regards
Ruihong
-----邮件原件-----
发件人: Mohammad Tariq [mailto:donta...@gmail.com]
发送时间: 2012年6月20日 15:26
收件人: flume-user@incubator.apache.org
主题: Re: How to configure "Taildir" in flume 1.x
Hello Shara,
There is no built-in taildir in Flume-NG..But it provides an
exec source that executes a given command and consumes the output. You
can use it to tail the files.Your config would be something like this
-
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /var/log/apache2/access.log
agent1.sources.tail.channels = MemoryChannel-2
......
Regards,
Mohammad Tariq
On Wed, Jun 20, 2012 at 12:49 PM, Shara Shi<shiruih...@dhgate.com> wrote:
Hi
Flume1.X does not support “Taildir” function in flume 0.9.X
How do I configure flume 1.X to meet the requirement like “taildir” in flume
0.9.X ?
Regards
Ruihong
