any chance you can leave it running for an extended period of time? 2014-10-24 18:12 GMT+02:00 Xavier Fustero <[email protected]>:
> Hi, > > I stopped rsyslogd and started it using valgrind. But as I said, if I > restart it then it seems to be working fine. Difficult to troubleshooting > as it happens randomly on the hundred servers we have. I haven not find a > clear pattern where I can reproduce it. > > The Valgrind report says everything looks good. I also run it again using > --leak-check=full --show-leak-kinds=all but got same positive report. > > Thanks Rainer, > Xavi > > ==20795== HEAP SUMMARY: > ==20795== in use at exit: 14,112 bytes in 181 blocks > ==20795== total heap usage: 796,711 allocs, 796,530 frees, 540,451,754 > bytes allocated > ==20795== > ==20795== LEAK SUMMARY: > ==20795== definitely lost: 0 bytes in 0 blocks > ==20795== indirectly lost: 0 bytes in 0 blocks > ==20795== possibly lost: 0 bytes in 0 blocks > ==20795== still reachable: 14,112 bytes in 181 blocks > ==20795== suppressed: 0 bytes in 0 blocks > ==20795== Reachable blocks (those to which a pointer was found) are not > shown. > ==20795== To see them, rerun with: --leak-check=full --show-leak-kinds=all > ==20795== > ==20795== For counts of detected and suppressed errors, rerun with: -v > ==20795== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 1 from 1) > > On 24 October 2014 17:20, Rainer Gerhards <[email protected]> > wrote: > > > 2014-10-24 16:25 GMT+02:00 Xavier Fustero <[email protected]>: > > > > > Hi all, > > > > > > we realized that our rsyslogs in production start consuming lot of > > memory. > > > This happens randomly and restarting the process it gets back to > normal. > > > Today I found one guy consuming 8.5gb! Note that I am talking about RES > > > memory, not VIRT. > > > > > > top -b -n 1 > > > > > > Mem: 17489776k total, 17107728k used, 382048k free, 60144k > buffers > > > Swap: 8388604k total, 6564100k used, 1824504k free, 427884k cached > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 12464 rs-daemo 20 0 759m 578m 3668 R 76 3.4 70:25.66 ruby > > > 29254 rs-daemo 20 0 727m 534m 5624 R 39 3.1 1:23.35 ruby > > > 6005 rs-daemo 20 0 808m 619m 3600 R 37 3.6 213:30.88 ruby > > > 6007 rs-daemo 20 0 810m 622m 3604 S 25 3.6 1102:55 ruby > > > 5039 rs-daemo 20 0 89228 26m 928 S 4 0.2 43:32.86 ruby > > > 4331 root 20 0 9112 1112 836 R 2 0.0 0:00.01 top > > > 5180 rs-daemo 20 0 89240 26m 916 S 2 0.2 10:07.45 ruby > > > 5275 rs-daemo 20 0 89128 26m 860 S 2 0.2 12:07.84 ruby > > > 6107 rs-daemo 20 0 107m 46m 864 S 2 0.3 20:01.43 ruby > > > 14392 syslog 20 0 15.2g 8.5g 820 S 2 50.9 237:18.47 rsyslogd > > > .... > > > > > > Not sure if this helps but looking at its threads I saw this: > > > > > > 6099 rs-daemo 20 0 89228 26m 916 S 0.3 0.2 10:14.67 ruby > > > 14397 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 95:50.32 rs:main > > Q:Reg > > > 14400 syslog 20 0 15.3g 8.5g 820 S 0.3 51.1 63:23.82 rs:action > 9 > > > que > > > > > > We are running rsyslog version 8.4.0 > > > > > > root@host:~# rsyslogd -v > > > > rsyslogd 8.4.0, compiled with: > > > > FEATURE_REGEXP: Yes > > > > GSSAPI Kerberos 5 support: No > > > > FEATURE_DEBUG (debug build, slow code): No > > > > 32bit Atomic operations supported: Yes > > > > 64bit Atomic operations supported: Yes > > > > memory allocator: system default > > > > Runtime Instrumentation (slow code): No > > > > uuid support: Yes > > > > Number of Bits in RainerScript integers: 32 (due to too-old > json-c > > > lib) > > > > > > > > See http://www.rsyslog.com for more information. > > > > > > > > > > This box is getting logs from several services, writing them locally > and > > > fwd to a central rsyslog server. The central rsyslog looks healthy and > > > nothing apparently is being queued in this rsyslog. We are using disk > > > assisted. Not sure on memory but nothing on disk for sure. I paste > below > > > few lines from rsyslog-stats: > > > > > > Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > > > > > > > Fri Oct 24 14:06:39 2014: imudp(*:514): submitted=0 > > > > Fri Oct 24 14:06:39 2014: resource-usage: utime=6252434752 > > > > stime=8003212169 maxrss=9156300 minflt=3878658 majflt=712 > inblock=50224 > > > > oublock=36923936 nvcsw=368075743 nivcsw=133053533 > > > > Fri Oct 24 14:06:39 2014: action 1 queue[DA]: size=0 enqueued=0 > full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=0 > > > > Fri Oct 24 14:06:39 2014: action 1 queue: size=0 enqueued=0 full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=0 > > > > Fri Oct 24 14:06:39 2014: action 9 queue[DA]: size=0 enqueued=0 > full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=0 > > > > Fri Oct 24 14:06:39 2014: action 9 queue: size=0 enqueued=101932930 > > > full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=902 > > > > Fri Oct 24 14:06:39 2014: action 10 queue[DA]: size=0 enqueued=0 > full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=0 > > > > Fri Oct 24 14:06:39 2014: action 10 queue: size=0 enqueued=1360512 > > full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=2375 > > > > Fri Oct 24 14:06:39 2014: main Q: size=0 enqueued=103293442 full=0 > > > > discarded.full=0 discarded.nf=0 maxqsize=1044 > > > > Fri Oct 24 14:06:39 2014: imudp(w0): called.recvmmsg=0 > called.recvmsg=0 > > > > msgs.received=0 > > > > > > > > > > so nothing seems queued so far. > > > > > > We are using the following template to fwd mssges: > > > # Templates for logging remotely > > > template(name="GroupApp" type="string" > > > string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% > > > %syslogtag%shard3/daemons:%msg%\n" > > > ) > > > > > > if $syslogfacility-text == 'local0' or $syslogfacility-text == 'local1' > > or > > > $syslogfacility-text == 'local2' then { > > > action(type="omrelp" target="OUR_SERVER" port="OUR_PORT" > > > template="GroupApp" > > > queue.filename="app_queue" > > > queue.type="linkedlist" > > > queue.spoolDirectory="/mnt/spool/rsyslog" > > > queue.highwatermark="8000" > > > queue.lowwatermark="6000" > > > queue.maxdiskspace="1g" > > > queue.timeoutenqueue="0" > > > queue.saveonshutdown="on" > > > queue.size="10000" ) > > > stop > > > } > > > > > > I haven't paste any debug file as if I restart it then the memory > > > consumption is back to normal. It is happening quite often since we > moved > > > from default Ubuntu 12.04 rsyslog5.8 to install rsyslog8.4.0. > > > > > > Anyone experienced similar issues? Any idea on how to troubleshooting > > this? > > > > > > > can you run it under valgrind, at least for a while, so that we can see > if > > valgrind reports any memory leaks? It is best to do this in the > foreground > > and without auto-backgrounding, just like this: > > > > $ sudo valgrind rsyslogd -n ...usual options... > > > > After a while, ctl-c out of it and valgrind emits a report. Note that > there > > are always some small "leaks", which are things not cleaned up because > > there is no need to. > > > > Rainer > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
