So indeed it is logstash causing the backup. If I do UDP, the backup doesn't happen - I haven't verified logstash is getting every single message, but it does seem to be getting the vast majority. If I do TCP in an action queue, that action queue starts backing up as the main queue did previously - it's getting some messages to logstash, but most seem to just be backing up in the queue. Restarting logstash doesn't seem to help at all - those queue files always seem to stay on disk once they're there.
I've been meaning to look in to bypassing logstash and go directly to ES from rsyslog, but I already have a good bit invested in the logstash rules for parsing a good deal of logs (LDAP, RADIUS, Apache, etc) and really didn't want to have to go through all that again. On Nov 12, 2013, at 11:33 AM, Dave Caplinger <[email protected]> wrote: > On Nov 8, 2013, at 4:27 PM, Damiano Verzulli <[email protected]> wrote: > >>> # Send logs to logstash for indexing *.* >>> @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat >> >> (BTW: in my case, my LOG were not sent to LogStash, but to a remote >> syslog. Anyway, problem rely on the transport: syslog over TCP) > ... >> ... LOTS of TCP open connection (from the OS point of view) but all in >> a "waiting" state (from central rsyslog point of view). > > I've run into a similar situation as well; that one boiled down to: the > output message rate (to the remote destination) could not keep up with the > input message rate. As a result, the output (action) queue filled, which > then caused the main queue to start filling, which ultimately resulted in the > main queue dropping messages once it was full. (Also, if the queue sizes are > large, this winds up introducing a lot of delay as messages wait to get to > the end of the main queue and then wait again to get to the end of the output > queue.) Enabling pstats output was very helpful in figuring out exactly what > was going on. Initially, I set pstats to write directly to it's own files > (bypassing the message queues), but once the situation was resolved, went > back to "regular" delivery since direct pstats files cannot be log-rotated. > >> Everything easly solved with: >> - replacing "@@" with "@" (as this means using UDP, with none of the >> overhead described above) or, in my case... >> - finding the original source of LOG (process running of the third >> machine) and simply killing it, so to stop the enormous flow of (useless) >> log. > > One thing that concerns me with your solution (switch from TCP to UDP) -- are > you actually receiving the logs at the remote destination now? I'm sure > rsyslog can blast UDP outbound very rapidly, but if you were in a backlogged > situation before, I'm concerned that you may have now made message drops > invisible. > > As for the original (Torrance Leggett's) issue sending to logstash; if the > issue is that logstash sometimes becomes unable to receive messages, it might > make more sense to set up that rsyslog *action* queue rather than the main > queue in Disk Assisted mode to help get you through intermittent issues like > this. (I recommend against using DA for the main queue at all.) This means > the main queue will complete it's work by writing to the local file hierarchy > via your templates, and only the remote copy of messages will [potentially] > get delayed or dropped. > > Then you can fix your logstash receiver when it breaks (or the elasticsearch > system that logstash may be feeding) and rsyslog will resume delivery of all > the messages logstash missed in the mean time. > > PS - If you really are feeding rsyslog -> logstash -> elasticsearch, consider > that rsyslog can be configured to feed the messages to elasticsearch directly > (in the same format logstash uses). Search the archives or web for recent > discussion on this topic. But again, you'll likely want to set up DA mode at > rsyslog's sending side action queue in order to weather any intermittent > receive-side issues. > > HTH, > > - Dave
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
