I thought I would give this a shot: https://github.com/mozilla-services/heka/pull/1486/
Any comments would be appreciated! On Wed, Apr 15, 2015 at 10:57 AM, Rob Miller <[email protected]> wrote: > On 04/13/2015 08:53 AM, M GS wrote: > >> Is it possible to start the LogstreamerInput without catching up old logs? >> > No, but it's becoming clear that we should maybe add this. I'm thinking a > boolean setting that, if true *and* there's no cursor file for the > logstream, will cause the processing to start at the tail rather than the > head of the stream. I've opened an issue for this: > https://github.com/mozilla-services/heka/issues/1482 > >> In my use case oldest_duration is not enough because the log is >> constantly being updated. >> > Not sure I understand what this means. Is it that even the current log > file is too big to process? If so, maybe you want to consider rotation your > logs a bit more often? > > One wrinkle in all of this is that we've realized there's a bug in the > oldest_duration code, causing it not to be honored; I've got it fixed on a > branch, am expecting to land that on the versions/0.9 branch today, so it > will go out in the 0.9.2 release that I'm hoping to make later today or > tomorrow. Maybe oldest_duration would be sufficient, if it were actually > working correctly. > >> The issue this causes is that parsing large >> log file attempts to inject too many messages when heka first starts up. >> I realize that we will end up loosing messages this way, but as it >> stands its very hard to deploy (ie: we have to rotate the logs manually, >> then start heka). >> >> Ideally what I do want is to parse all our logs, but I can't do this on >> a production machine anyway. What I would rather do is have heka start >> fresh each time when in production. >> > Hopefully the oldest_duration fix in 0.9.2 will help you at least a bit, > otherwise we'll work on getting #1482 implemented. Sorry for the headache. > > -r > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
