https://github.com/mozilla-services/heka/issues/1855
On 02/05/2016 03:53 PM, Rob Miller wrote:
Hrm. Most likely sitch is that there's a race condition where Heka is
trying to find it's new location in the log stream but it's still
operating with the prior (i.e. no longer valid) set of file names. That
would explain both why it didn't find the new location in the stream
(b/c that has a file name that didn't exist prior) and why the journal
is pointing at a file that no longer exists.
I'll open an issue for this.
-r
On 02/03/2016 11:05 AM, Dai Shi wrote:
> Hello,
>
> I've only been using heka for a couple months, and this is my first time
> posting to this mailing list. I've been running into the following issue
> with logstreamer input, and was hoping someone on here might point me in
> the right direction.
>
> Every so often, when logrotate rotates logs, the logstreamer input
> appears to start tailing the oldest log, which gets deleted during the
> rotate, and then it will just get stuck there and not make any progress.
>
> Here is my logstreamer config:
>
> [MongoTailer]
> type = "LogstreamerInput"
> log_directory = "/data/log/mongo"
> journal_directory = "/data/appdata/heka"
> # expect files named mongod.<rset>.log[-YYYYMMDD]
> file_match =
> 'mongod\.(?P<Rset>\S+)\.log-?((?P<Year>\d{4})(?P<Month>\d{2})(?P<Day>\d{2}))?'
>
> differentiator = ["all.", "Rset"]
> priority = ["Year", "Month", "Day"]
>
> # make the log ending in .log the most recent
> [MongoTailer.translation.Year]
> missing = 9999
>
> Running heka-logstreamer on the config shows what I expect:
>
> Logstream name: [all.venues0]
> Files: 8 (printing oldest to newest)
> /data/log/mongo/mongod.venues0.log-20160121
> /data/log/mongo/mongod.venues0.log-20160122
> /data/log/mongo/mongod.venues0.log-20160123
> /data/log/mongo/mongod.venues0.log-20160124
> /data/log/mongo/mongod.venues0.log-20160125
> /data/log/mongo/mongod.venues0.log-20160126
> /data/log/mongo/mongod.venues0.log-20160127
> /data/log/mongo/mongod.venues0.log
>
> During normal operation, looking at the journal file also shows what I
> expect:
>
>
{"seek":79749937,"file_name":"/data/log/mongo/mongod.venues0.log","last_hash":"be40779a9aaa7371ee8a53fc448ba10d7c084fe3"}
>
>
> But, sometimes when logrotate does rotation, the journal file will then
> show:
>
>
{"seek":55436986,"file_name":"/data/log/mongo/mongod.venues0.log-20160120","last_hash":"ad99fe64c5f6cba6a2a197dfa14cc10c583cf712"}
>
>
> The file it appears to tail no longer exists, and it will just sit there
> until I bounce hekad. I'm not sure why it would go from tailing the
> newest file to tailing the oldest file. After bouncing it will go back
> through all the old logs again.
>
> Any ideas what could be happening?
>
>
> _______________________________________________
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>
_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
