On Fri, 18 Mar 2016, Thomas D. wrote:
Hi,
David Lang wrote:
I'm turning off async writes and will report on if this stabilizes the memory
utilization.
to clarify, the timeline was approximatly
3am bump dynafilebuffersize up to 800, shrink the iobuffersize to 64K, 4am
change to async, 5am shrink iobuffersize to 32K, 6am shrink iobuffersize to 16K,
10am deploy new version, 2pm disable async, 2:30 comment out a third dynafile
action, 3pm split perserver_q ruleset.
prior to turning on async, perserver_q was clearly the limiting thread, with
async on, it's utilization dropped to ~40% (throughput increased slightly, with
the dyna_files thread being the bottleneck), but the thread count skyrocketed
from 17 to 1500+ and memory utilization climed (normal memory used by rsyslog
pre-async was ~4%, I saw it above 80% and it hit OOM a half dozen times)
when I deployed the new version, throughput suffered significantly and there
still appeared to be a memory leak, but a much smaller/slower one, after 4 hours
ory utilization had climbed to ~30% vs OOMing about every hour.
turning off async on the new version stopped the memory leak, no signficant
difference in throughput.
splitting the perserver_q ruleset into two rulesets boosted throughput back up
to what I was getting with 8.17
Can you share the related omfile settings like "dynaFileCacheSize",
"flushInterval", "ioBufferSize" you are using when experiencing the problem?
And how many async omfile actions do you actual have? Are you able to
sum it up (you said your are using dynfiles...)?
I had three actions, I turned one of them off (in the interest of improving
performance), each one is creating ~500 files with ziplevel=9
before I turned off async, these were both in the perserver_q ruleset.
ruleset(name="perserver_q" queue.type="FixedArray" queue.dequeuebatchsize =
"500"){
action(type="omfile" template="json" dynafile="perserverjson"
dynafilecachesize="800" ziplevel="9" flushinterval="60" name="perserverjson"
ioBufferSize="16k" )
}
ruleset(name="perserver_q2" queue.type="FixedArray" queue.dequeuebatchsize =
"500"){
action(type="omfile" template="raw" dynafile="perserverdebug"
dynafilecachesize="800" ziplevel="9" flushinterval="60" name="perserverdebug"
ioBufferSize="16k" )
}
Once I split the perserver_q ruleset, I have one omprog program and the thread
below both bumping up against 100% cpu. I don't see an easy way to split this
thread :-(
ruleset(name="dyna_files" queue.type="FixedArray" queue.dequeuebatchsize =
"500") {
set $!trusted!severity = $syslogseverity-text;
set $!trusted!facility = $syslogfacility-text;
set $!trusted!programname = $programname;
foreach ($.tag in $!event.tags) do {
set $.custommessage = $.tag;
#if not ($.tag startswith 'ASA-' or $.tag == ["trash"]) then {
if not ($.tag startswith 'ASA-') then {
if ($.tag == 'json') then {
set $.custommessage = $programname;
}
action(type="omfile" name="tagfiles" dynafile="tagfiles" template="std"
dynafilecachesize="400" ioBufferSize="16K" flushinterval="10")
action(type="omfile" name="tagparsedfiles" dynafile="tagparsedfiles"
template="json" dynafilecachesize="400" ioBufferSize="16K" flushinterval="10")
}
}
/var/log/messages-tags;manual
call perserver_q
call perserver_q2
}
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
