Hi all, we realized that our rsyslogs in production start consuming lot of memory. This happens randomly and restarting the process it gets back to normal. Today I found one guy consuming 8.5gb! Note that I am talking about RES memory, not VIRT.
top -b -n 1 Mem: 17489776k total, 17107728k used, 382048k free, 60144k buffers Swap: 8388604k total, 6564100k used, 1824504k free, 427884k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12464 rs-daemo 20 0 759m 578m 3668 R 76 3.4 70:25.66 ruby 29254 rs-daemo 20 0 727m 534m 5624 R 39 3.1 1:23.35 ruby 6005 rs-daemo 20 0 808m 619m 3600 R 37 3.6 213:30.88 ruby 6007 rs-daemo 20 0 810m 622m 3604 S 25 3.6 1102:55 ruby 5039 rs-daemo 20 0 89228 26m 928 S 4 0.2 43:32.86 ruby 4331 root 20 0 9112 1112 836 R 2 0.0 0:00.01 top 5180 rs-daemo 20 0 89240 26m 916 S 2 0.2 10:07.45 ruby 5275 rs-daemo 20 0 89128 26m 860 S 2 0.2 12:07.84 ruby 6107 rs-daemo 20 0 107m 46m 864 S 2 0.3 20:01.43 ruby 14392 syslog 20 0 15.2g 8.5g 820 S 2 50.9 237:18.47 rsyslogd .... Not sure if this helps but looking at its threads I saw this: 6099 rs-daemo 20 0 89228 26m 916 S 0.3 0.2 10:14.67 ruby 14397 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 95:50.32 rs:main Q:Reg 14400 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 63:23.82 rs:action 9 que We are running rsyslog version 8.4.0 root@host:~# rsyslogd -v > rsyslogd 8.4.0, compiled with: > FEATURE_REGEXP: Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: Yes > memory allocator: system default > Runtime Instrumentation (slow code): No > uuid support: Yes > Number of Bits in RainerScript integers: 32 (due to too-old json-c lib) > > See http://www.rsyslog.com for more information. > This box is getting logs from several services, writing them locally and fwd to a central rsyslog server. The central rsyslog looks healthy and nothing apparently is being queued in this rsyslog. We are using disk assisted. Not sure on memory but nothing on disk for sure. I paste below few lines from rsyslog-stats: Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > Fri Oct 24 14:06:39 2014: resource-usage: utime=6252434752 > stime=8003212169 maxrss=9156300 minflt=3878658 majflt=712 inblock=50224 > oublock=36923936 nvcsw=368075743 nivcsw=133053533 > Fri Oct 24 14:06:39 2014: action 1 queue[DA]: size=0 enqueued=0 full=0 > discarded.full=0 discarded.nf=0 maxqsize=0 > Fri Oct 24 14:06:39 2014: action 1 queue: size=0 enqueued=0 full=0 > discarded.full=0 discarded.nf=0 maxqsize=0 > Fri Oct 24 14:06:39 2014: action 9 queue[DA]: size=0 enqueued=0 full=0 > discarded.full=0 discarded.nf=0 maxqsize=0 > Fri Oct 24 14:06:39 2014: action 9 queue: size=0 enqueued=101932930 full=0 > discarded.full=0 discarded.nf=0 maxqsize=902 > Fri Oct 24 14:06:39 2014: action 10 queue[DA]: size=0 enqueued=0 full=0 > discarded.full=0 discarded.nf=0 maxqsize=0 > Fri Oct 24 14:06:39 2014: action 10 queue: size=0 enqueued=1360512 full=0 > discarded.full=0 discarded.nf=0 maxqsize=2375 > Fri Oct 24 14:06:39 2014: main Q: size=0 enqueued=103293442 full=0 > discarded.full=0 discarded.nf=0 maxqsize=1044 > Fri Oct 24 14:06:39 2014: imudp(w0): called.recvmmsg=0 called.recvmsg=0 > msgs.received=0 > so nothing seems queued so far. We are using the following template to fwd mssges: # Templates for logging remotely template(name="GroupApp" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%shard3/daemons:%msg%\n" ) if $syslogfacility-text == 'local0' or $syslogfacility-text == 'local1' or $syslogfacility-text == 'local2' then { action(type="omrelp" target="OUR_SERVER" port="OUR_PORT" template="GroupApp" queue.filename="app_queue" queue.type="linkedlist" queue.spoolDirectory="/mnt/spool/rsyslog" queue.highwatermark="8000" queue.lowwatermark="6000" queue.maxdiskspace="1g" queue.timeoutenqueue="0" queue.saveonshutdown="on" queue.size="10000" ) stop } I haven't paste any debug file as if I restart it then the memory consumption is back to normal. It is happening quite often since we moved from default Ubuntu 12.04 rsyslog5.8 to install rsyslog8.4.0. Anyone experienced similar issues? Any idea on how to troubleshooting this? Thanks a log, Xavi _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
