what modules do you use? I've been running 8.15-pre git builds for a while
(tracking down the segfaults that Rainer just traced down to json-c) and have
had no memory issues.
But I'm not using imfile or omelasticsearch.
David Lang
On Wed, 16 Dec 2015, Ciprian Hacman wrote:
Date: Wed, 16 Dec 2015 18:00:48 +0200
From: Ciprian Hacman <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Rsyslog using huge amounts of memory
Not sure if you saw, but provided output from Valgrind with bout inotify
and polling.
To me it seems that in both cases the leak can be seen.
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Dec 16, 2015 at 5:23 PM, Rainer Gerhards <[email protected]>
wrote:
2015-12-16 16:21 GMT+01:00 Ciprian Hacman <[email protected]>:
Not sure how easy is to reproduce on your side.
If you need help narrowing the leak to a certain commit or release, I
could
help.
OK, I thought you could just switch the imfile mode. So I suggest to
open a bug tracker, I guess I can look into it after the holiday
period.
Rainer
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Dec 16, 2015 at 4:22 PM, Rainer Gerhards <
[email protected]>
wrote:
2015-12-16 15:10 GMT+01:00 Ciprian Hacman <[email protected]
:
Done. Can you check the gist again?
yup. It now contains better info, but I wonder where these leaks stem
from. One suspect is inotify polling mode, which we didn't really test
well for quite a while...
Rainer
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Dec 16, 2015 at 3:40 PM, Rainer Gerhards <
[email protected]>
wrote:
2015-12-16 14:16 GMT+01:00 Ciprian Hacman <
[email protected]
:
Done! I updated the output in the same place:
https://gist.github.com/hakman/44afddaf4eb67cda28c6
mhhh... less clear than I had hoped...
Can you switch imfile to use inotify mode and retry?
Rainer
Thanks for looking into this so fast!
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Dec 16, 2015 at 2:53 PM, Rainer Gerhards <
[email protected]>
wrote:
2015-12-16 13:48 GMT+01:00 Ciprian Hacman <
[email protected]
:
Hi,
I upgraded a server to Rsyslog 8.15 last night and today the
process
was
using almost 200MB of RAM (raising steadily).
Tried running the process in Valgrind to see if I get an idea
about
what
happens, but wasn't that much help for me.
That's because debug symbols are unloaded at module unload time.
This
makes valgrind stacktraces unusable. Nevertheless, the information
looks very promising.
Can you build rsyslog yourself for that box? All we need is the
--enable-valgrind option, which will essentially remove the module
unloads and make the stacktrace usable.
Rainer
If someone has better debugging skills, I pasted the output
here.
Not
sure
if I let it run enough or leave it running longer.
https://gist.github.com/hakman/44afddaf4eb67cda28c6
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Dec 15, 2015 at 2:41 PM, David Lang <[email protected]>
wrote:
On Tue, 15 Dec 2015, Ciprian Hacman wrote:
Hi David,
maxMessageSize="10000"
queue.size="10000" - main queue
queue.size="10000" - elasticsearch queue
Based on my calculations this brings me to a max of 200MB of
memory,
maybe
a little more depending on how maxMessageSize is calculated.
I read logs from a file and push them to Elasticsearch (on the
same
network), so TCP is the only possibility. This server has a
very
simple
setup.
If I don't find the reason for this issue, I might have to go
implement
the
forwarding to a central location and push to Elasticsearch
from
there.
There are advantages to sending things to a central server.
it's one place to queue data, so you can either allocate more
ram,
or go
to disk as needed without impacting other workloads.
it's more efficient, the central server is more likely to have
larger
batches of data to feed to ES, and ES only needs to be running
one
thread
processing inbound data
while it is one more point to have to look at, I think it
simplifies
troubleshooting as all the communication to ES (and therefor
all
the
errors
for such communication) happen in one place instead of
distributed.
anyway, let's see how things look with 8.15
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
