Hi We use the TailSource in flume to tail our log files. Whenever the log files reach 1gb in size they are rolled. We are experiencing a problem where flume cannot keep up with the log write rate and eventually missing an entire file because it rolls over before flume notices. I think this happens because flume doesn't detect the roll until it finishes tailing the current file. I am think a simple solution to start would be to include a timestamp in the log files and use a tailDir so that every file is seen. My only concern is we will end up with more and more tail threads as the writing gets further and further ahead.
Just wondering if anyone else experienced and dealt with this problem? Thanks
