-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > [...] However, recently I’ve had a problem where the server stops > writing out virtually all such messages and the main message queue > fills and [....]
I spent several hours, today, to troubleshoot _exactly_ this problem and, if your production environment is similar to mine, chance are that the root-cause is similar. In my case (where there are several tens of servers logging locally as well as on a central rsyslog-server, and with a total amount of +/- 20GB of daily log produced/archived), the "root" of the problem was exactly this line: > # Send logs to logstash for indexing *.* > @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat (BTW: in my case, my LOG were not sent to LogStash, but to a remote syslog. Anyway, problem rely on the transport: syslog over TCP) What happened is that a third host (neither the one that stopped logging nor the remote syslog), due to a bug in a process, started bombing the log-server (with UDP messages) with something like thousands of messages per second. As such, the remote/central rsyslog simply started... being a bit "busy", so not to be able to correctly process all the incoming request. As our first system (the one with logging suddenly stopping) were using TCP, here is that sending LOG messages to the central-server via TCP, means LOTS of TCP open connection (from the OS point of view) but all in a "waiting" state (from central rsyslog point of view). Unfortunatly, the more the logging messages generated => the more opened socket => the sooner stopping logging. Everything easly solved with: - - replacing "@@" with "@" (as this means using UDP, with none of the overhead described above) or, in my case... - - finding the original source of LOG (process running of the third machine) and simply killing it, so to stop the enormous flow of (useless) log. HTH. Bye, DV - -- Damiano Verzulli e-mail: [email protected] - --- possible?ok:while(!possible){open_mindedness++} - --- "Technical people tend to fall into two categories: Specialists and Generalists. The Specialist learns more and more about a narrower and narrower field, until he eventually, in the limit, knows everything about nothing. The Generalist learns less and less about a wider and wider field, until eventually he knows nothing about everything." - William Stucke - AfrISPA http://elists.isoc.org/mailman/private/pubsoft/2007-December/001935.html -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlJ9ZUQACgkQcwT9fsMT4SxO5QCeLQVRu0p0SZJ9RyoyI2QuZVnu KnUAnjsPKUjJU98lZUZj/FMyo6LZfMGW =CFsQ -----END PGP SIGNATURE----- _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
