Huge file is better for file tailing adaptor actually. FileTailing adaptor checks every 2 seconds to see if file has rotated. If the rotating frequency is high, like rotating every seconds. Then there may be a chance of losing track of correct offset in the files. Rotated log every couple minutes with large size, helps FileTailing adaptor to track offset in the files correctly. As far as throughput, it can stream up to 10MB/sec from our performance test.
regards, Eric On Thu, Dec 22, 2011 at 6:44 PM, AD <[email protected]> wrote: > hey, > > does anyone know if there are any throughput issues with FileTailing > adaptors? If i am writing huge files is there risk of missing lines > (especially if the files are getting rotated). I am just trying to decide > if running map/reduce on a rotated log every X minutes is more safe than > FileTail if the log writes get massively high (thousands/sec) > > Cheers, > AD
