Ok, this shows that the queues at this point in time are pretty empty, but at
some point they had filled up. If they all filled up at the same time (a
reasonable possibility), you had ~23K messages queued at one point.
do you use fixedarray or linkedlist as your queue type?
do you do much parsing of the logs? I don't know exactly how the memory is
accounted for, but if you have a message size of 10K, and have it parsed into
many messages, your $! variables are going to be more than 10K by themselves. I
don't know if $msg and $rawmsg are separate copies in ram or just pointers into
the original message. The ES module would create a large string of the entire
batch it's trying to process before sending it to the ES server (what's your
batch size? you would have hit it right after the outage where you were unable
to deliver to ES)?
it still seems a stretch for 23K messages to use 6G of ram. that's ~250K/message
when you have a max message size of 10K (if I've done my math right). If we were
talking a factor of 3x or 4x I would shrug and put it down to plain overhead,
but 25x seems a bit much :-)
Let's see if Rainer has any other thoughts when he wakes up in the next few
hours.
David Lang
On Tue, 15 Dec 2015, Ciprian Hacman wrote:
Date: Tue, 15 Dec 2015 08:43:52 +0200
From: Ciprian Hacman <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Rsyslog using huge amounts of memory
Hi Peter,
Just checked and impstats were actually enabled. Here is the sample before
restart:
Dec 14 09:25:09 host1 rsyslogd-pstats: omelasticsearch:
origin=omelasticsearch submitted=198115631 failed.http=260
failed.httprequests=2 failed.es=58720
Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm-direct: origin=core.action
processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0
Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct: origin=core.action
processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0
Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm: origin=core.action
processed=1040122 failed=0 suspended=0 suspended.duration=0 resumed=0
Dec 14 09:25:09 host1 rsyslogd-pstats: test-es: origin=core.action
processed=1039926 failed=0 suspended=0 suspended.duration=0 resumed=0
Dec 14 09:25:09 host1 rsyslogd-pstats: resource-usage: origin=impstats
utime=5859695897 stime=206002475 maxrss=6587196 minflt=15802757
majflt=19279 inblock=3838232 oublock=2238104 nvcsw=5667006 nivcsw=596991
Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct queue:
origin=core.queue size=75 enqueued=197076777 full=1742 discarded.full=1423
discarded.nf=0 maxqsize=10000
Dec 14 09:25:09 host1 rsyslogd-pstats: test-es queue: origin=core.queue
size=0 enqueued=1039926 full=0 discarded.full=0 discarded.nf=0 maxqsize=2888
Dec 14 09:25:09 host1 rsyslogd-pstats: main Q: origin=core.queue size=27
enqueued=198705203 full=0 discarded.full=0 discarded.nf=0 maxqsize=9765
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Dec 15, 2015 at 7:57 AM, Peter Portante <[email protected]>
wrote:
On Tue, Dec 15, 2015 at 12:30 AM, Ciprian Hacman <
[email protected]> wrote:
Hi David,
maxMessageSize="10000"
queue.size="10000" - main queue
queue.size="10000" - elasticsearch queue
Based on my calculations this brings me to a max of 200MB of memory,
maybe
a little more depending on how maxMessageSize is calculated.
I read logs from a file and push them to Elasticsearch (on the same
network), so TCP is the only possibility. This server has a very simple
setup.
If I don't find the reason for this issue, I might have to go implement
the
forwarding to a central location and push to Elasticsearch from there.
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote:
what is your maxmessagesize and your max queue size? rsyslog will use
up
to maxmessagesize*maxqueuesize ram to buffer messages if they can't be
delivered.
you probably want to set these values smaller rather than setting
something up to kill rsyslog when it gets large.
What is the transport you use to deliver the logs from these systems?
I like to setup log redundant log relay servers in each datacenter and
then have all the systems log to these relays via UDP. UDP is reliable
over
a local network, but if there is a problem with the receiving system,
it
will go ahead and loose logs rather than affecting the sending system.
David Lang
On Mon, 14 Dec 2015, Ciprian Hacman wrote:
Hi David,
Yes, killing Rsyslog is a last resort, but for most people I think
shipping
logs is a secondary function on a server. Would prefer that Rsyslog
doesn't
interfere with other apps.
I usually enable impstats, though on these particular server the
queues
are
really tiny so that it doesn't use that much memory. I would expect
some
memory usage fluctuations when Elasticsearch doesn't respond, but
nothing
as extreme as using 6GB of memory.
If changes in 8.15 don't help, I think I have to spend a few hours
trying
to reproduce this.
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote:
On Mon, 14 Dec 2015, Ciprian Hacman wrote:
Hi,
We are noticing some Rsyslog instances that push about 500MB of logs
daily
to an Elasticsearch instance, so not that much. We noticed one of
the
Rsyslog processes using about 6GB of RAM. Usually this is less than
1MB.
I plan on debugging this in the next few days, but wanted to see
also
if
there is a good idea to add a RSS limit (doable in Upstart and
Systemd)
and
kill / restart Rsyslog when it gets into such a situation.
killing/restarting rsyslog is a last resort. large memory usage
usually
means that you have lots of logs that aren't delivered and are
sitting
in a
queue somewhere.
do you have impstats configured?
Ciprian, are you going to enable impstats? I'd be curious to know what I
shows.
Thanks, -peter
if not, it's a _really_ good idea to
configure it and have it write either directly to a file (log
rotation
of
this file is a bit of an issue) or to it's own ruleset. either way
means
that a blockage in normal log processing will not affect the pstats
logs.
These logs will show you if you have queues building up and where.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
