what is your maxmessagesize and your max queue size? rsyslog will use up to
maxmessagesize*maxqueuesize ram to buffer messages if they can't be delivered.
you probably want to set these values smaller rather than setting something up
to kill rsyslog when it gets large.
What is the transport you use to deliver the logs from these systems?
I like to setup log redundant log relay servers in each datacenter and then have
all the systems log to these relays via UDP. UDP is reliable over a local
network, but if there is a problem with the receiving system, it will go ahead
and loose logs rather than affecting the sending system.
David Lang
On Mon, 14 Dec 2015, Ciprian Hacman wrote:
Hi David,
Yes, killing Rsyslog is a last resort, but for most people I think shipping
logs is a secondary function on a server. Would prefer that Rsyslog doesn't
interfere with other apps.
I usually enable impstats, though on these particular server the queues are
really tiny so that it doesn't use that much memory. I would expect some
memory usage fluctuations when Elasticsearch doesn't respond, but nothing
as extreme as using 6GB of memory.
If changes in 8.15 don't help, I think I have to spend a few hours trying
to reproduce this.
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote:
On Mon, 14 Dec 2015, Ciprian Hacman wrote:
Hi,
We are noticing some Rsyslog instances that push about 500MB of logs daily
to an Elasticsearch instance, so not that much. We noticed one of the
Rsyslog processes using about 6GB of RAM. Usually this is less than 1MB.
I plan on debugging this in the next few days, but wanted to see also if
there is a good idea to add a RSS limit (doable in Upstart and Systemd)
and
kill / restart Rsyslog when it gets into such a situation.
killing/restarting rsyslog is a last resort. large memory usage usually
means that you have lots of logs that aren't delivered and are sitting in a
queue somewhere.
do you have impstats configured? if not, it's a _really_ good idea to
configure it and have it write either directly to a file (log rotation of
this file is a bit of an issue) or to it's own ruleset. either way means
that a blockage in normal log processing will not affect the pstats logs.
These logs will show you if you have queues building up and where.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
