Thanks. I fully suspect the bottleneck upstream and am adding more instance of logstash/shippers behind my haproxy instance. If there wasn't a bottleneck, would rsyslog push more out on its own or do I need to increase batch/thread/queue? to get more out assuming a steady state? Will it try to push/process as much out from queues/cache every time and backoff if it can't?
Is there a point of no return (no improvement) in tweaking the knobs? Doug On Mon, Aug 4, 2014 at 3:02 PM, David Lang <[email protected]> wrote: > well, it's clear that you are getting new requests FAR faster than you are > processing them > > Mon Aug 4 13:14:16 2014: imuxsock: submitted=3 ratelimit.discarded=0 > ratelimit.numratelimiters=2 > Mon Aug 4 13:14:16 2014: action 1: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 2: processed=603 failed=0 > Mon Aug 4 13:14:16 2014: action 3: processed=547 failed=0 > Mon Aug 4 13:14:16 2014: action 4: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 5: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 6: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 7: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 8: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: action 9: processed=0 failed=0 > Mon Aug 4 13:14:16 2014: logstashforwarder: processed=270878 failed=0 > Mon Aug 4 13:14:16 2014: imptcp(*/10514/IPv4): submitted=270859 > Mon Aug 4 13:14:16 2014: imptcp(*/10514/IPv6): submitted=0 > Mon Aug 4 13:14:16 2014: logstashforwarder[DA]: size=73726973 > enqueued=114807 full=0 discarded.full=0 discarded.nf=0 maxqsize=73756802 > Mon Aug 4 13:14:16 2014: logstashforwarder: size=147 enqueued=270878 > full=0 discarded.full=0 discarded.nf=0 maxqsize=9770 > Mon Aug 4 13:14:16 2014: main Q: size=0 enqueued=270878 full=0 > discarded.full=0 discarded.nf=0 maxqsize=31209 > > > > Mon Aug 4 13:15:16 2014: imuxsock: submitted=10 ratelimit.discarded=0 > ratelimit.numratelimiters=6 > Mon Aug 4 13:15:16 2014: action 1: processed=0 failed=0 > Mon Aug 4 13:15:16 2014: action 2: processed=1877 failed=0 > Mon Aug 4 13:15:16 2014: action 3: processed=592 failed=0 > Mon Aug 4 13:15:16 2014: action 4: processed=4 failed=0 > Mon Aug 4 13:15:16 2014: action 5: processed=2 failed=0 > Mon Aug 4 13:15:16 2014: action 6: processed=0 failed=0 > Mon Aug 4 13:15:16 2014: action 7: processed=0 failed=0 > Mon Aug 4 13:15:16 2014: action 8: processed=0 failed=0 > Mon Aug 4 13:15:16 2014: action 9: processed=0 failed=0 > Mon Aug 4 13:15:16 2014: logstashforwarder: processed=694102 failed=0 > Mon Aug 4 13:15:16 2014: imptcp(*/10514/IPv4): submitted=696044 > Mon Aug 4 13:15:16 2014: imptcp(*/10514/IPv6): submitted=0 > Mon Aug 4 13:15:16 2014: logstashforwarder[DA]: size=73817861 > enqueued=317479 full=0 discarded.full=0 discarded.nf=0 maxqsize=73817861 > Mon Aug 4 13:15:16 2014: logstashforwarder: size=1392 enqueued=694130 > full=0 discarded.full=0 discarded.nf=0 maxqsize=9770 > Mon Aug 4 13:15:16 2014: main Q: size=4150 enqueued=696078 full=0 > discarded.full=0 discarded.nf=0 maxqsize=31209 > > if you look at the queue sizes, in this timeframe you fell WAY behind, you > received more messages more than you processed (the difference in the cache > sizes for the logstashforwarder Q, logstashforwarder[DA] and main Q size > stats). It looks like you fell behind by >100k messages > > So this looks to me like the logstash instance just isn't able to keep up, > can you look at the data there? > > also, it would be good to restart this with the DA cache files removed, > putting messages into the DA cache files does cost performance. > > At this data volume, I'd also suggest changing the impstats time down to > something like 10 seconds so that the numbers don't get too big. > > David Lang > > > On Mon, 4 Aug 2014, Doug McClure wrote: > > Date: Mon, 4 Aug 2014 14:49:38 -0400 >> From: Doug McClure <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Finding the holy grail tuning setting... >> >> >> I appreciate it - I desire an objective approach this challenge! >> >> Attached is a fresh impstats file. Appreciate any interpretation advice >> and tuning actions. >> >> Doug >> >> >> On Mon, Aug 4, 2014 at 1:05 PM, David Lang <[email protected]> wrote: >> >> On Mon, 4 Aug 2014, Doug McClure wrote: >>> >>> I've read, re-read and read again everything I can find out there on >>> >>>> queues, options, etc. and still feel I don't really know what I'm doing >>>> other than haphazardly changing one or more settings hoping to get more >>>> data through/out of rsyslog. >>>> >>>> I'm growing about one 1GB DA cache file every 10 min or so and I can't >>>> seem >>>> to increase the processing to clear them up. I probably clear one for >>>> every 2-4 new ones that are created. >>>> >>>> What's the best setting to focus on to increase DA queue file >>>> processing? >>>> I've taken dequeuebatchsize from as low as 100 or 1000 (which everything >>>> seems to talk about) to as high as 100,000 or more and I can't seem to >>>> hit >>>> a sweet spot. I have varied threads up to 200. Queue size up to 1G or >>>> 500M >>>> or 200K. >>>> >>>> What are the rules of thumb here - change X watch Y until you get to >>>> some >>>> ceiling and you need to add more system resources or other upstream? >>>> >>>> >>> well, rather than focusing on the DA queue handling, let's try and figure >>> out what's slow and causing things to queue >>> >>> have you configured impstats? configure it to log to a file, and log >>> fairly frequently and we should be able to see what action is holding >>> things up. Once we know that we can work to figure out how to solve that >>> bottlneck. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
