> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Timur I. Bakeyev > Sent: Wednesday, November 07, 2012 2:27 PM > To: rsyslog-users > Cc: Michael Biebl > Subject: Re: [rsyslog] Rsyslog freezes the box when can't send logs > over TCP > > Hi, Rainer! > > Of course you know internals of rsyslog better, so can't argue with you > here. But as for the end user such behavior for me looks undesired. > What > I'd expect: > > 1. All local(imuxsock) go to the local log files. > 2. All remote messages would be spooled in an $ActionQueueFileName > file. > 3. Nothing would block
Yup - so why not configure it that way? Actually, very reliable logging is a tough consulting job. It's nothing you do in the lunch break. > > I also found this old discussion: > > http://www.mail-archive.com/[email protected]/msg01330.html > > So you tried to resolve it in 3.x series, but, apparently the problem > is > still there. There is NO Problem. There may have been one in v3, I don't remember - too long ago. > > DO I understand correctly that the reason is in the fact that Output > queue > for local files and remote TCP is shared, so until confirmation from > both > backends is received the message isn't discarded? > > Is there a way to spit those queues? > > Or, can you suggest the configuration that does, what I've described? > Any > change if TCP would be replaced with RELP? > You need to read up the in-depth description on how queues work. You need to know when it is OK to lose messages. Then, you need to configure the system to do that. The root cause is not the protocol, but your desired to have "nothing lost" but not thought what should happen if you run out of resources. Again, this is a though question and doing it right is a real consulting job. Rainer Rainer > With regards, > Timur Bakeyev. > > On Tue, Nov 6, 2012 at 3:37 PM, Rainer Gerhards > <[email protected]>wrote: > > > Long story short: the problem is with your config. You configured > rsyslog > > to never lose any message, at some point it runs out of queue space, > then > > the inputs begin to block, so the log socket blocks and finally the > apps > > that write to the log socket. > > There are many options to control under which circumstances you > permit > > message loss, and you need to make sure you know what you want. > > The current versions also have lower enq timeout, which can kind of > solve > > this config issue, at least for systems that are not super-busy. Note > that > > it is not really an issue, but may be intended behavior in some > > environments (e.g. banking, when you absolutely need to log). > > > > Rainer > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Timur I. Bakeyev > > > Sent: Tuesday, November 06, 2012 3:26 PM > > > To: [email protected] > > > Cc: Michael Biebl > > > Subject: [rsyslog] Rsyslog freezes the box when can't send logs > over > > > TCP > > > > > > Hi all! > > > > > > Recently we experienced a "nice" lock up of our web nodes that do > > > remote TCP logging to the centralized syslog server. > > > > > > Central rsyslog server was shut down due the maintenance for few > > > hours. Very soon first Web servers with nginx/php-fpm started to > fail > > > with timeouts and later access to the nodes themselves over ssh > > > stopped working, making boxes unaccessible. With the iLO it was > > > possible to log to the consoles of the nodes, but no problems were > > > spotted - memory, CPU, TCP sockets footprints were low, everything > > > looked normal. > > > > > > Educated guess that it may be related to the rsyslog, as it was the > > > only change in the servers configuration on that day and killing > > > rsyslogd on the nodes brought machines back to normal. > > > > > > A bit of googling brought up this story: > > > > > > http://blog.bitbucket.org/2012/01/12/follow-up-on-our-downtime- > last- > > > week/ > > > > > > as well as a last year conversation on this ML: > > > > > > http://lists.adiscon.net/pipermail/rsyslog/2011-October/013944.html > > > > > > as well as a bug report in RedHat Bugzilla: > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=519201 > > > > > > Switching to UDP transport instead of TCP cured the problem and > > > running simple test with: > > > > > > # for each n in `seq 1 1000`; do echo $n; echo "Syslog test $n" | > > > logger; done > > > > > > showed that with TCP remote logging enabled and absent remote log > > > server after first 250-300 messages all the following messages were > > > sent to syslog very slowly, with the 1-2sec delay each. > > > > > > All nodes are running Debian 6(squeeze) with: > > > > > > rsyslogd 5.8.11, compiled with: > > > FEATURE_REGEXP: Yes > > > FEATURE_LARGEFILE: No > > > GSSAPI Kerberos 5 support: Yes > > > FEATURE_DEBUG (debug build, slow code): No > > > 32bit Atomic operations supported: Yes > > > 64bit Atomic operations supported: Yes > > > Runtime Instrumentation (slow code): No > > > > > > And configuration: > > > > > > $MaxMessageSize 8k > > > $ModLoad imuxsock # provides support for local system logging > > > $ModLoad imklog # provides kernel logging support > > > > > > # Store PID of the process in the log > > > $SystemLogUsePIDFromSystem on > > > # Rate limit for imuxsock > > > $SystemLogRateLimitInterval 1 > > > $SystemLogRateLimitBurst 500 > > > > > > $WorkDirectory /var/spool/rsyslog > > > > > > $ModLoad imfile > > > $InputFilePollInterval 5 > > > > > > $InputFileName /var/log/nginx/access.log > > > $InputFilePersistStateInterval 100 > > > $InputFileTag nginx/access: > > > $InputFileStateFile nginx_access_log_state > > > $InputFileFacility local7 > > > $InputFileSeverity notice > > > $InputRunFileMonitor > > > > > > $ActionQueueType LinkedList # enable a > > > separate queue for this action > > > $ActionQueueFileName remote # set file > name, > > > also enables disk mode > > > $ActionResumeRetryCount -1 # infinite > > > retries on insert failure > > > $ActionQueueSaveOnShutdown on > > > *.* @@10.0.0.200 > > > > > > I went through ChangeLog on the site for the versions of rsyslogd > > > following 5.8.11, but didn't notice any fix to that problem. > Neither > > > there is an open ticket in bugzilla. > > > > > > Interesting enough that RedHat claims they fixed the problem in > version > > > 3.22: > > > > > > http://rhn.redhat.com/errata/RHBA-2010-0213.html > > > > > > Can you point me to the version where this problem is fixed? Or, if > > > it's not fixed yet - can it be done? > > > > > > BTW, not sure how this would behave with REPL protocol instead of > > > TCP... > > > > > > with best regards, > > > Timur Bakeyev. > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > > if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
