2015-12-15 8:24 GMT+01:00 David Lang <[email protected]>: > Ok, this shows that the queues at this point in time are pretty empty, but > at some point they had filled up. If they all filled up at the same time (a > reasonable possibility), you had ~23K messages queued at one point. > > do you use fixedarray or linkedlist as your queue type? > > do you do much parsing of the logs? I don't know exactly how the memory is > accounted for, but if you have a message size of 10K, and have it parsed > into many messages, your $! variables are going to be more than 10K by > themselves. I don't know if $msg and $rawmsg are separate copies in ram or > just pointers into the original message. The ES module would create a large > string of the entire batch it's trying to process before sending it to the > ES server (what's your batch size? you would have hit it right after the > outage where you were unable to deliver to ES)? > > it still seems a stretch for 23K messages to use 6G of ram. that's > ~250K/message when you have a max message size of 10K (if I've done my math > right). If we were talking a factor of 3x or 4x I would shrug and put it > down to plain overhead, but 25x seems a bit much :-) > > Let's see if Rainer has any other thoughts when he wakes up in the next few > hours.
I would prefer if we look into 8.15, simply because I *know* it has some fixes for memleaks that very seldom show up. If the problem persists, it makes sense to debug, but I'd prefer not to hunt potentially fixed bugs... Rainer > > David Lang > > On Tue, 15 Dec 2015, Ciprian Hacman wrote: > >> Date: Tue, 15 Dec 2015 08:43:52 +0200 >> From: Ciprian Hacman <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Rsyslog using huge amounts of memory >> >> >> Hi Peter, >> >> Just checked and impstats were actually enabled. Here is the sample before >> restart: >> >> Dec 14 09:25:09 host1 rsyslogd-pstats: omelasticsearch: >> origin=omelasticsearch submitted=198115631 failed.http=260 >> failed.httprequests=2 failed.es=58720 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm-direct: >> origin=core.action >> processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct: origin=core.action >> processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm: origin=core.action >> processed=1040122 failed=0 suspended=0 suspended.duration=0 resumed=0 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es: origin=core.action >> processed=1039926 failed=0 suspended=0 suspended.duration=0 resumed=0 >> Dec 14 09:25:09 host1 rsyslogd-pstats: resource-usage: origin=impstats >> utime=5859695897 stime=206002475 maxrss=6587196 minflt=15802757 >> majflt=19279 inblock=3838232 oublock=2238104 nvcsw=5667006 nivcsw=596991 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct queue: >> origin=core.queue size=75 enqueued=197076777 full=1742 discarded.full=1423 >> discarded.nf=0 maxqsize=10000 >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es queue: origin=core.queue >> size=0 enqueued=1039926 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=2888 >> Dec 14 09:25:09 host1 rsyslogd-pstats: main Q: origin=core.queue size=27 >> enqueued=198705203 full=0 discarded.full=0 discarded.nf=0 maxqsize=9765 >> >> >> Ciprian >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Tue, Dec 15, 2015 at 7:57 AM, Peter Portante >> <[email protected]> >> wrote: >> >>> On Tue, Dec 15, 2015 at 12:30 AM, Ciprian Hacman < >>> [email protected]> wrote: >>> >>>> Hi David, >>>> >>>> maxMessageSize="10000" >>>> queue.size="10000" - main queue >>>> queue.size="10000" - elasticsearch queue >>>> >>>> Based on my calculations this brings me to a max of 200MB of memory, >>> >>> maybe >>>> >>>> a little more depending on how maxMessageSize is calculated. >>>> >>>> I read logs from a file and push them to Elasticsearch (on the same >>>> network), so TCP is the only possibility. This server has a very simple >>>> setup. >>>> >>>> If I don't find the reason for this issue, I might have to go implement >>> >>> the >>>> >>>> forwarding to a central location and push to Elasticsearch from there. >>>> >>>> Thanks, >>>> Ciprian >>>> >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote: >>>> >>>>> what is your maxmessagesize and your max queue size? rsyslog will use >>> >>> up >>>>> >>>>> to maxmessagesize*maxqueuesize ram to buffer messages if they can't be >>>>> delivered. >>>>> >>>>> you probably want to set these values smaller rather than setting >>>>> something up to kill rsyslog when it gets large. >>>>> >>>>> What is the transport you use to deliver the logs from these systems? >>>>> >>>>> I like to setup log redundant log relay servers in each datacenter and >>>>> then have all the systems log to these relays via UDP. UDP is reliable >>>> >>>> over >>>>> >>>>> a local network, but if there is a problem with the receiving system, >>> >>> it >>>>> >>>>> will go ahead and loose logs rather than affecting the sending system. >>>>> >>>>> David Lang >>>>> >>>>> >>>>> >>>>> On Mon, 14 Dec 2015, Ciprian Hacman wrote: >>>>> >>>>> Hi David, >>>>>> >>>>>> >>>>>> Yes, killing Rsyslog is a last resort, but for most people I think >>>>>> shipping >>>>>> logs is a secondary function on a server. Would prefer that Rsyslog >>>>>> doesn't >>>>>> interfere with other apps. >>>>>> >>>>>> I usually enable impstats, though on these particular server the >>> >>> queues >>>>>> >>>>>> are >>>>>> really tiny so that it doesn't use that much memory. I would expect >>> >>> some >>>>>> >>>>>> memory usage fluctuations when Elasticsearch doesn't respond, but >>>> >>>> nothing >>>>>> >>>>>> as extreme as using 6GB of memory. >>>>>> >>>>>> If changes in 8.15 don't help, I think I have to spend a few hours >>>> >>>> trying >>>>>> >>>>>> to reproduce this. >>>>>> >>>>>> Thanks, >>>>>> Ciprian >>>>>> >>>>>> -- >>>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>> >>>>>> On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Mon, 14 Dec 2015, Ciprian Hacman wrote: >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>>> >>>>>>>> We are noticing some Rsyslog instances that push about 500MB of logs >>>>>>>> daily >>>>>>>> to an Elasticsearch instance, so not that much. We noticed one of >>> >>> the >>>>>>>> >>>>>>>> Rsyslog processes using about 6GB of RAM. Usually this is less than >>>> >>>> 1MB. >>>>>>>> >>>>>>>> >>>>>>>> I plan on debugging this in the next few days, but wanted to see >>> >>> also >>>> >>>> if >>>>>>>> >>>>>>>> there is a good idea to add a RSS limit (doable in Upstart and >>>> >>>> Systemd) >>>>>>>> >>>>>>>> and >>>>>>>> kill / restart Rsyslog when it gets into such a situation. >>>>>>>> >>>>>>>> >>>>>>> killing/restarting rsyslog is a last resort. large memory usage >>> >>> usually >>>>>>> >>>>>>> means that you have lots of logs that aren't delivered and are >>> >>> sitting >>>>>>> >>>>>>> in a >>>>>>> queue somewhere. >>>>>>> >>>>>>> do you have impstats configured? >>>> >>>> >>> >>> Ciprian, are you going to enable impstats? I'd be curious to know what I >>> shows. >>> >>> Thanks, -peter >>> >>> >>> >>>> if not, it's a _really_ good idea to >>>>>>> >>>>>>> configure it and have it write either directly to a file (log >>> >>> rotation >>>> >>>> of >>>>>>> >>>>>>> this file is a bit of an issue) or to it's own ruleset. either way >>>> >>>> means >>>>>>> >>>>>>> that a blockage in normal log processing will not affect the pstats >>>> >>>> logs. >>>>>>> >>>>>>> These logs will show you if you have queues building up and where. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad >>>>>>> >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> >>> you >>>>>>> >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad >>>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>> >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad >>>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
