Hi David, maxMessageSize="10000" queue.size="10000" - main queue queue.size="10000" - elasticsearch queue
Based on my calculations this brings me to a max of 200MB of memory, maybe a little more depending on how maxMessageSize is calculated. I read logs from a file and push them to Elasticsearch (on the same network), so TCP is the only possibility. This server has a very simple setup. If I don't find the reason for this issue, I might have to go implement the forwarding to a central location and push to Elasticsearch from there. Thanks, Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote: > what is your maxmessagesize and your max queue size? rsyslog will use up > to maxmessagesize*maxqueuesize ram to buffer messages if they can't be > delivered. > > you probably want to set these values smaller rather than setting > something up to kill rsyslog when it gets large. > > What is the transport you use to deliver the logs from these systems? > > I like to setup log redundant log relay servers in each datacenter and > then have all the systems log to these relays via UDP. UDP is reliable over > a local network, but if there is a problem with the receiving system, it > will go ahead and loose logs rather than affecting the sending system. > > David Lang > > > > On Mon, 14 Dec 2015, Ciprian Hacman wrote: > > Hi David, >> >> Yes, killing Rsyslog is a last resort, but for most people I think >> shipping >> logs is a secondary function on a server. Would prefer that Rsyslog >> doesn't >> interfere with other apps. >> >> I usually enable impstats, though on these particular server the queues >> are >> really tiny so that it doesn't use that much memory. I would expect some >> memory usage fluctuations when Elasticsearch doesn't respond, but nothing >> as extreme as using 6GB of memory. >> >> If changes in 8.15 don't help, I think I have to spend a few hours trying >> to reproduce this. >> >> Thanks, >> Ciprian >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote: >> >> On Mon, 14 Dec 2015, Ciprian Hacman wrote: >>> >>> Hi, >>> >>>> >>>> We are noticing some Rsyslog instances that push about 500MB of logs >>>> daily >>>> to an Elasticsearch instance, so not that much. We noticed one of the >>>> Rsyslog processes using about 6GB of RAM. Usually this is less than 1MB. >>>> >>>> I plan on debugging this in the next few days, but wanted to see also if >>>> there is a good idea to add a RSS limit (doable in Upstart and Systemd) >>>> and >>>> kill / restart Rsyslog when it gets into such a situation. >>>> >>>> >>> killing/restarting rsyslog is a last resort. large memory usage usually >>> means that you have lots of logs that aren't delivered and are sitting >>> in a >>> queue somewhere. >>> >>> do you have impstats configured? if not, it's a _really_ good idea to >>> configure it and have it write either directly to a file (log rotation of >>> this file is a bit of an issue) or to it's own ruleset. either way means >>> that a blockage in normal log processing will not affect the pstats logs. >>> These logs will show you if you have queues building up and where. >>> >>> David Lang >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
