May be I got it wrong. I thought it's pretty common use case to send continuous feed of logs as data is written to the logs. How do other people design with Flume NG when using logs like apache, web server or other kinds of logs?
On Mon, Apr 2, 2012 at 8:34 PM, Juhani Connolly < juhani_conno...@cyberagent.co.jp> wrote: > Easiest way is to make some kind of daemon process that regularly stat the > file, checking for changes, and stores a position within the file up to > which has been read/sent. > Some things you might want to watch out for: > - Shrinking file generally means a log roll > - Change in inode means the old file has been moved and a new one has been > made with the same name. You might want to follow the old one, or at least > check if it has any unflushed changes left(since the concept of inodes is > platform specific, you can't do this with all languages) > > You may want to look at the TailSource code from Flume OG for some general > ideas. > Just to warn you however, that tracking all the possible things that can > happen to a file can be pretty messy. > > > On 04/03/2012 07:07 AM, Mohit Anchlia wrote: > >> I just ran avro client and saw that it streamed entire file and exited. >> How can I design a client that continuously keeps forwarding the changes to >> the log file? >> > >
