Hi
This might be not enough yet to diagnose, but here's an spoiler. We have
managed to review configuration and improve memory footprint, but still
growing over time and ends being killed by oom:
module(load="impstats" log.file="/data/stats.log")
syslog.=debug /data/rsyslog-stats
global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
)
input(
type="imrelp"
ruleset="relp"
)
ruleset(
name="relp"
queue.filename="relp"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
) {
action(
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error #to write error.log
stop
}
action(
type="mmnormalize"
#actually it's doing nothing, cause liblognorm issues.
)
set some variables;
if $!app != $!app then {
call unknown #to write unk.log
stop
}
#this line include files with if's to re-set variables
$IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf
call clean
set a few variables;
action(
type="omelasticsearch"
)
}
#include other rulesets to handle other things, like error or unknown
$IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf
Wed Jan 18 09:47:57 2017: global: origin=dynstats
Wed Jan 18 09:47:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: omelasticsearch: origin=omelasticsearch
submitted=235 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 09:47:57 2017: json: origin=core.action processed=235
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: norm: origin=core.action processed=235
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: elastic: origin=core.action processed=235
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: imrelp[20514]: origin=imrelp submitted=235
Wed Jan 18 09:47:57 2017: resource-usage: origin=impstats utime=168000
stime=124000 maxrss=7828 minflt=1383 majflt=0 inblock=10 oublock=16
nvcsw=1628 nivcsw=44
Wed Jan 18 09:47:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:47:57 2017: relp: origin=core.queue size=0 enqueued=235
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:47:57 2017: main Q: origin=core.queue size=9 enqueued=12
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 09:52:57 2017: global: origin=dynstats
Wed Jan 18 09:52:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: omelasticsearch: origin=omelasticsearch
submitted=418 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 09:52:57 2017: json: origin=core.action processed=418
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: norm: origin=core.action processed=418
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: elastic: origin=core.action processed=418
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: imrelp[20514]: origin=imrelp submitted=418
Wed Jan 18 09:52:57 2017: resource-usage: origin=impstats utime=236000
stime=232000 maxrss=7828 minflt=1385 majflt=0 inblock=10 oublock=18
nvcsw=2710 nivcsw=59
Wed Jan 18 09:52:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:52:57 2017: relp: origin=core.queue size=0 enqueued=418
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:52:57 2017: main Q: origin=core.queue size=0 enqueued=25
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 09:57:57 2017: global: origin=dynstats
Wed Jan 18 09:57:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: omelasticsearch: origin=omelasticsearch
submitted=574 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 09:57:57 2017: json: origin=core.action processed=574
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: norm: origin=core.action processed=574
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: elastic: origin=core.action processed=574
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: imrelp[20514]: origin=imrelp submitted=574
Wed Jan 18 09:57:57 2017: resource-usage: origin=impstats utime=332000
stime=288000 maxrss=7828 minflt=1385 majflt=0 inblock=10 oublock=21
nvcsw=3634 nivcsw=67
Wed Jan 18 09:57:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:57:57 2017: relp: origin=core.queue size=0 enqueued=574
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:57:57 2017: main Q: origin=core.queue size=0 enqueued=38
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:02:57 2017: global: origin=dynstats
Wed Jan 18 10:02:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: omelasticsearch: origin=omelasticsearch
submitted=755 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:02:57 2017: json: origin=core.action processed=755
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: norm: origin=core.action processed=755
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: elastic: origin=core.action processed=755
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:02:57 2017: imrelp[20514]: origin=imrelp submitted=755
Wed Jan 18 10:02:57 2017: resource-usage: origin=impstats utime=416000
stime=376000 maxrss=7828 minflt=1388 majflt=0 inblock=10 oublock=24
nvcsw=4689 nivcsw=75
Wed Jan 18 10:02:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:02:57 2017: relp: origin=core.queue size=0 enqueued=755
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:02:57 2017: main Q: origin=core.queue size=0 enqueued=51
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:07:57 2017: global: origin=dynstats
Wed Jan 18 10:07:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: omelasticsearch: origin=omelasticsearch
submitted=934 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:07:57 2017: json: origin=core.action processed=934
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: norm: origin=core.action processed=934
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: elastic: origin=core.action processed=934
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:07:57 2017: imrelp[20514]: origin=imrelp submitted=934
Wed Jan 18 10:07:57 2017: resource-usage: origin=impstats utime=520000
stime=448000 maxrss=7828 minflt=1398 majflt=0 inblock=10 oublock=27
nvcsw=5741 nivcsw=86
Wed Jan 18 10:07:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:07:57 2017: relp: origin=core.queue size=0 enqueued=934
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:07:57 2017: main Q: origin=core.queue size=0 enqueued=64
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Wed Jan 18 10:12:57 2017: global: origin=dynstats
Wed Jan 18 10:12:57 2017: action 0: origin=core.action processed=0
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: omelasticsearch: origin=omelasticsearch
submitted=1114 failed.http=0 failed.httprequests=0 failed.checkConn=0
failed.es=0
Wed Jan 18 10:12:57 2017: json: origin=core.action processed=1114
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: norm: origin=core.action processed=1114
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: elastic: origin=core.action processed=1114
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: error: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: unk: origin=core.action processed=0 failed=0
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:12:57 2017: imrelp[20514]: origin=imrelp submitted=1114
Wed Jan 18 10:12:57 2017: resource-usage: origin=impstats utime=608000
stime=528000 maxrss=7828 minflt=1411 majflt=0 inblock=10 oublock=30
nvcsw=6812 nivcsw=93
Wed Jan 18 10:12:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:12:57 2017: relp: origin=core.queue size=0 enqueued=1114
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:12:57 2017: main Q: origin=core.queue size=0 enqueued=77
full=0 discarded.full=0 discarded.nf=0 maxqsize=9
Feedback much appreciated.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
