Hi, pls send the config for your sink. You create one file in a directory, and after that file has reached 1GB you reset that?
best, Alex -- Alexander Lorenz http://mapredit.blogspot.com On Feb 23, 2012, at 10:23 AM, Cameron Gandevia wrote: > Hi > > We use the TailSource in flume to tail our log files. Whenever the log files > reach 1gb in size they are rolled. We are experiencing a problem where flume > cannot keep up with the log write rate and eventually missing an entire file > because it rolls over before flume notices. I think this happens because > flume doesn't detect the roll until it finishes tailing the current file. I > am think a simple solution to start would be to include a timestamp in the > log files and use a tailDir so that every file is seen. My only concern is we > will end up with more and more tail threads as the writing gets further and > further ahead. > > Just wondering if anyone else experienced and dealt with this problem? > > Thanks > >
