2014-10-24 16:25 GMT+02:00 Xavier Fustero <[email protected]>: > Hi all, > > we realized that our rsyslogs in production start consuming lot of memory. > This happens randomly and restarting the process it gets back to normal. > Today I found one guy consuming 8.5gb! Note that I am talking about RES > memory, not VIRT. > > top -b -n 1 > > Mem: 17489776k total, 17107728k used, 382048k free, 60144k buffers > Swap: 8388604k total, 6564100k used, 1824504k free, 427884k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12464 rs-daemo 20 0 759m 578m 3668 R 76 3.4 70:25.66 ruby > 29254 rs-daemo 20 0 727m 534m 5624 R 39 3.1 1:23.35 ruby > 6005 rs-daemo 20 0 808m 619m 3600 R 37 3.6 213:30.88 ruby > 6007 rs-daemo 20 0 810m 622m 3604 S 25 3.6 1102:55 ruby > 5039 rs-daemo 20 0 89228 26m 928 S 4 0.2 43:32.86 ruby > 4331 root 20 0 9112 1112 836 R 2 0.0 0:00.01 top > 5180 rs-daemo 20 0 89240 26m 916 S 2 0.2 10:07.45 ruby > 5275 rs-daemo 20 0 89128 26m 860 S 2 0.2 12:07.84 ruby > 6107 rs-daemo 20 0 107m 46m 864 S 2 0.3 20:01.43 ruby > 14392 syslog 20 0 15.2g 8.5g 820 S 2 50.9 237:18.47 rsyslogd > .... > > Not sure if this helps but looking at its threads I saw this: > > 6099 rs-daemo 20 0 89228 26m 916 S 0.3 0.2 10:14.67 ruby > 14397 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 95:50.32 rs:main Q:Reg > 14400 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 63:23.82 rs:action 9 > que > > We are running rsyslog version 8.4.0 > > root@host:~# rsyslogd -v > > rsyslogd 8.4.0, compiled with: > > FEATURE_REGEXP: Yes > > GSSAPI Kerberos 5 support: No > > FEATURE_DEBUG (debug build, slow code): No > > 32bit Atomic operations supported: Yes > > 64bit Atomic operations supported: Yes > > memory allocator: system default > > Runtime Instrumentation (slow code): No > > uuid support: Yes > > Number of Bits in RainerScript integers: 32 (due to too-old json-c > lib) > > > > See http://www.rsyslog.com for more information. > > > > This box is getting logs from several services, writing them locally and > fwd to a central rsyslog server. The central rsyslog looks healthy and > nothing apparently is being queued in this rsyslog. We are using disk > assisted. Not sure on memory but nothing on disk for sure. I paste below > few lines from rsyslog-stats: > > Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > > > Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > > Fri Oct 24 14:06:39 2014: resource-usage: utime=6252434752 > > stime=8003212169 maxrss=9156300 minflt=3878658 majflt=712 inblock=50224 > > oublock=36923936 nvcsw=368075743 nivcsw=133053533 > > Fri Oct 24 14:06:39 2014: action 1 queue[DA]: size=0 enqueued=0 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=0 > > Fri Oct 24 14:06:39 2014: action 1 queue: size=0 enqueued=0 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=0 > > Fri Oct 24 14:06:39 2014: action 9 queue[DA]: size=0 enqueued=0 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=0 > > Fri Oct 24 14:06:39 2014: action 9 queue: size=0 enqueued=101932930 > full=0 > > discarded.full=0 discarded.nf=0 maxqsize=902 > > Fri Oct 24 14:06:39 2014: action 10 queue[DA]: size=0 enqueued=0 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=0 > > Fri Oct 24 14:06:39 2014: action 10 queue: size=0 enqueued=1360512 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=2375 > > Fri Oct 24 14:06:39 2014: main Q: size=0 enqueued=103293442 full=0 > > discarded.full=0 discarded.nf=0 maxqsize=1044 > > Fri Oct 24 14:06:39 2014: imudp(w0): called.recvmmsg=0 called.recvmsg=0 > > msgs.received=0 > > > > so nothing seems queued so far. > > We are using the following template to fwd mssges: > # Templates for logging remotely > template(name="GroupApp" type="string" > string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% > %syslogtag%shard3/daemons:%msg%\n" > ) > > if $syslogfacility-text == 'local0' or $syslogfacility-text == 'local1' or > $syslogfacility-text == 'local2' then { > action(type="omrelp" target="OUR_SERVER" port="OUR_PORT" > template="GroupApp" > queue.filename="app_queue" > queue.type="linkedlist" > queue.spoolDirectory="/mnt/spool/rsyslog" > queue.highwatermark="8000" > queue.lowwatermark="6000" > queue.maxdiskspace="1g" > queue.timeoutenqueue="0" > queue.saveonshutdown="on" > queue.size="10000" ) > stop > } > > I haven't paste any debug file as if I restart it then the memory > consumption is back to normal. It is happening quite often since we moved > from default Ubuntu 12.04 rsyslog5.8 to install rsyslog8.4.0. > > Anyone experienced similar issues? Any idea on how to troubleshooting this? >
can you run it under valgrind, at least for a while, so that we can see if valgrind reports any memory leaks? It is best to do this in the foreground and without auto-backgrounding, just like this: $ sudo valgrind rsyslogd -n ...usual options... After a while, ctl-c out of it and valgrind emits a report. Note that there are always some small "leaks", which are things not cleaned up because there is no need to. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
