The [DA] queue is actually the Disk-Assist mode. Think of it as another queue stuck on the end of your existing (in-memory) queue. If the regular output queue becomes full (because the remote destination is unavailable or too slow) then rsyslog will start spooling the overflow messages to disk rather than discarding them.
So stats that show the [DA] enqueueing indicate that it's hit that situation (as you described, the remote receiver couldn't keep up with the volume). Starting DA mode incurs a significant performance penalty, so you don't want to trigger it too early or stay in that mode longer than necessary. So some parameters you can adjust: 1) increase the queue.size of the customerstream queue; the maxqsize of 965 suggests that you may be using the default size of only 1000 messages. You can probably easily to 10x or more depending on the RAM available in the system. 2) if you increase queue.size, you should also increase queue.discardmark to 90-95% of the new queue.size value. 3) also, you should increase queue.highwatermark -- this is the size of the memory queue that will trigger writing to DA files. 4) increase the queue.lowwatermark number; the default is very low. You want to get back out of DA mode as quickly as possible, because when in DA mode, all new messages get written to disk (which is slow). Better for performance to run from RAM as much as possible and enter/leave DA mode repeatedly throughout the day than be stuck there. - Dave On May 6, 2014, at 9:55 AM, Michael Hart <[email protected]> wrote: > I dug into this some more and while the explanations are excellent (thanks!) > I don’t quite understand what I’m seeing. > > For background, I have a ruleset defined with a name “customerstream” that > does some simple pre-processing (templates, save to local disk) and then > forwards logs to logstash (action shown below). rsyslog v7.4.5 on Ubuntu > 12.04LTS. > > I know I’m queueing data, the spool directory (/var/spool/rsyslog) has at > time of writing 560Mb of files, about 550 files at 1Mb each, with current > timestamps. These are normal log lines, less than 1K each. I also know I’m > queueing because the downstream system is running flat out until late at > night when my logging volume slows down, this is a performance problem > unrelated to rsyslog. > > When I spit out my stats to file, I see two lines for that ruleset: > > May 6 14:42:09 redacted rsyslogd-pstats: @cee: > {"name":"customerstream[DA]","size":25692,"enqueued":66180,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":26844} > May 6 14:42:09 redacted rsyslogd-pstats: @cee: > {"name":"customerstream","size":421,"enqueued":133538,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":965} > > The second line is what I’m sending to statsd, and obviously is not correct. > The first one, with name “customerstream[DA]” is where I’m confused… for two > reasons. First when I restart rsyslog it starts at zero, even though all the > files in /var/spool/rsyslog are still there and AFAIK I’m not losing data. > The second… where does the [DA] come from and what does it mean? > > Here’s the same two lines immediately after restart (and it doesn’t climb > back to the original size): > > May 6 14:45:32 redacted rsyslogd-pstats: @cee: > {"name":"customerstream[DA]","size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0} > May 6 14:45:32 redacted rsyslogd-pstats: @cee: > {"name":"customerstream","size":0,"enqueued":2379,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":489} > > So my questions: > 1. What does the [DA] mean? Or, why do I see two lines for a ruleset? > 2. Why does the size go down to zero after restart and not show the full > queue size? > 3. Am I losing data on restarts? > > For reference the ruleset in question has the following action: > > action(type="omfwd" > name="customerstream" > target="localhost" > port="5544" > protocol="tcp" > template="LongTagForwardFormat" > queue.filename="customerstream" > queue.maxdiskspace="1073741824" > queue.saveonshutdown="on" > queue.type="LinkedList" > queue.discardmark="1000000" > queue.discardseverity="4" > queue.timeoutenqueue="0" > action.resumeretrycount="-1" > action.resumeinterval="30" > ) > > thanks! > mike > > -- > Michael Hart > Arctic Wolf Networks > M: 226-388-4773 > > > > > > On May 2, 2014, at 11:28 AM, Dave Caplinger > <[email protected]<mailto:[email protected]>> wrote: > > On May 1, 2014, at 4:15 PM, David Lang <[email protected]<mailto:[email protected]>> > wrote: > > On Thu, 1 May 2014, Dave Caplinger wrote: > > This specific line doesn't say so, but the implication is that all 110 > messages that entered the queue also successfully left the queue. > > not quite, 110 - 28 = 82 have successfully left the queue > > Whoops; you're right of course! > > - Dave > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
