Hi, If there was such a script I could launch when rsyslog get stuck, while waiting our V8 upgrade, that would be perfect !
Yes I'm sending logs directly to logstash using omrelp, we have 40 rsyslog instances sending logs to 3 logstash through a TCP LoadBalancer. I do not know if logstash is the bottleneck here, because our log rate output is pretty stable and the problem occurs from time to time (not because of burst as I thought previously) It looks like rsyslog is "kind of loosing" its connection to logstash after a certain time of inactivity. I can see (using netstat) that rsyslog is "connected" to logstash over a tcp connection on 5001 port, but no messages are sent over this socket. Could it have something to do with tcprebindinterval stated here ? www.rsyslog.com/load-balancing-for-rsyslog/ ? I've just discover this parameter in the doc I did some archeology in the ML, and I think V5 omrelp module does not support tcprebindinterval (based on this thread http://lists.adiscon.net/pipermail/rsyslog/2013-October/034451.html ) Thank you all for doing this community work! On 8 July 2015 at 19:17, Rainer Gerhards <[email protected]> wrote: > 2015-07-08 19:12 GMT+02:00 David Lang <[email protected]>: > > If you cannot loose any logs, then when you run out of disk space and > memory > > queue space your systems will stop working and you won't even be able to > > login to them (because doing so attempts to create a log entry) > > > > Also, using Disk Assisted Queues means that you have some log entries on > > disk and others in memory. When you restart rsyslog, the ones in memory > are > > going to be lost (because they can't be written to disk) > > > > So the biggest thing you need to do is to look at where the logs are > going > > and try to make that fast enough to keep up. If you are delivering logs > to > > logstash, what is logstash doing with them (sending them to ElasticSearch > > would be my guess, but are they manipulated first or sent elsewhere as > > well?) Rsyslog may be able to deliver directly to those destinations, > > bypassing the bottleneck of logstash > > > > Yes, there are ways to get rsyslog to read the queue files, I'd have to > hunt > > in the archives, but IIRC there is a utility that will create the qi > files > > so that rsyslog will notice them the next time it starts. I'd have to > hunt > > the list archives to find references to how to do it. > > Out of my head: it's a perl script, I think in ./tools ... > qi_recover.pl? If not, search for *.pl > > Rainer > > > > David Lang > > > > > > > > On Wed, 8 Jul 2015, Nicolas Guyomar wrote: > > > >> Hi everyone, > >> > >> Unfortunately I cannot loose any log, because they all are access.log > with > >> the same severity so I can't juste say "discard the lower level" > >> > >> Let's say my problem could be solved by upgrading to the latest Rsyslog > >> stable version (planned for this summer), can I "replay" the log flushed > >> into queue file in my work directory ? > >> > >> When my rsyslog V5 instance is stucked with its ActionQueueMaxDiskSpace > >> reached, restarting has no effect. I'd like to maybe save the queue file > >> to > >> some other directory, and copy old queue file in the work directory so > >> that > >> Rsyslog send them to logstash. > >> > >> > >> > >> > >> On 30 June 2015 at 11:41, Radu Gheorghe <[email protected]> > >> wrote: > >> > >>> Hi Nicolas, > >>> > >>> Unfortunately, I'm not aware of any specific issue here. But there are > >>> some > >>> options regarding discarding messages when the queue exceeds a certain > >>> size > >>> (look for DiscardMark and DiscardSeverity): > >>> > >>> > >>> > http://www.rsyslog.com/doc/v5-stable/configuration/action/index.html#action-queue-specific-configuration-statements > >>> > >>> Maybe you can find a workaround that way (I assume you can discard > >>> everything after you hit a certain limit). > >>> > >>> Best regards, > >>> Radu > >>> > >>> -- > >>> Performance Monitoring * Log Analytics * Search Analytics > >>> Solr & Elasticsearch Support * http://sematext.com/ > >>> > >>> On Tue, Jun 30, 2015 at 12:32 PM, Nicolas Guyomar < > >>> [email protected] > >>>> > >>>> wrote: > >>> > >>> > >>>> Hi, > >>>> > >>>> Upgrade to V8 is planned for this year, but in the meantime I thought > I > >>>> could find a way to maybe discard messages when rsyslog has no more > >>>> space > >>>> left in its work directory (which see. > >>>> Losing some messages during burst period is acceptable for us, but > being > >>>> forced to manually delete and restart is more complicated. > >>>> > >>>> I hoped the problem was some sort of misconfiguration on my side, or > >>> > >>> maybe > >>>> > >>>> a know issue using omrelp with logstash relp input. > >>>> > >>>> > >>>> > >>>> > >>>> On 30 June 2015 at 09:55, Radu Gheorghe <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi Nicolas, > >>>>> > >>>>> I have some vague memories about nasty bugs in disk-assisted queues > >>> > >>> that > >>>>> > >>>>> were fixed in the last few years. RELP modules surely have changed as > >>>> > >>>> well. > >>>>> > >>>>> Can you try with the latest stable (8.10 I think) and see if it > helps? > >>>> > >>>> Even > >>>>> > >>>>> if it doesn't, I'm pretty sure the fix will come in the 8.x branch > >>>> > >>>> because > >>>>> > >>>>> it sounds pretty serious. > >>>>> > >>>>> Best regards, > >>>>> Radu > >>>>> > >>>>> -- > >>>>> Performance Monitoring * Log Analytics * Search Analytics > >>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>> > >>>>> On Tue, Jun 30, 2015 at 10:42 AM, Nicolas Guyomar < > >>>>> [email protected] > >>>>>> > >>>>>> wrote: > >>>>> > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> I've got a simple question on disk assisted queue behaviour, it > could > >>>> > >>>> be > >>>>>> > >>>>>> trivial, but I can't find an answer on the internet. > >>>>>> > >>>>>> I'm using rsyslog V5 to forward nginx access log to some logstash > >>>>> > >>>>> instances > >>>>>> > >>>>>> using omrelp > >>>>>> > >>>>>> Sometimes, because of activity burst, rsyslog flush onto disk 200 > 1Mo > >>>>> > >>>>> files > >>>>>> > >>>>>> (which is the expected behaviour), but then stays stuck, no more > >>>> > >>>> messages > >>>>>> > >>>>>> are sent to logstash. > >>>>>> I have to delete state files as well as queue files so that rsyslog > >>>> > >>>> start > >>>>>> > >>>>>> sending messages to logstash again. > >>>>>> Restarting rsyslog without deleting those files has no effect. > >>>>>> > >>>>>> Here is my rsyslog config in case I missed something > >>>>>> > >>>>>> $RuleSet nginxRuleSet > >>>>>> $RulesetCreateMainQueue on > >>>>>> > >>>>>> $WorkDirectory /tmp > >>>>>> $ActionQueueFileName queue-nginx > >>>>>> $ActionQueueMaxDiskSpace 200m > >>>>>> $ActionQueueSaveOnShutdown on > >>>>>> $ActionQueueType LinkedList > >>>>>> $ActionQueueSize 540000 > >>>>>> $ActionResumeRetryCount -1 > >>>>>> *.* :omrelp:<%= @lblog %>:5001;erableTmpl > >>>>>> & ~ > >>>>>> > >>>>>> > >>>>>> $InputUDPServerBindRuleset nginxRuleSet > >>>>>> $UDPServerRun 515 > >>>>>> > >>>>>> Is it a known behaviour ? > >>>>>> > >>>>>> Thank you for any help one could provide > >>>>>> > >>>>>> Nicolas > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad > >>>>>> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>> > >>> you > >>>>>> > >>>>>> DON'T LIKE THAT. > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> > >>> myriad > >>>>> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
